System and method for concurrently conducting cause-and-effect experiments on content effectiveness and adjusting content distribution to optimize business objectives

ABSTRACT

The present invention is directed to systems, articles, and computer-implemented methods for assessing effectiveness of communication content and optimizing content distribution to enhance business objectives. Embodiments of the present invention are directed to computer-implemented methods for a computer-implemented method, comprising conducting an experiment using experimental content to determine effectiveness of communication content and executing, while conducting the experiment, a machine learning routine (MLR) using MLR content to enhance an effectiveness metric.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.12/651,650, filed Jan. 4, 2010, now allowed which claims the benefit ofU.S. Provisional Patent Application No. 61/143,060, filed Jan. 7, 2009,the disclosure of which is incorporated by reference herein in itsentirety.

RELATED APPLICATIONS

This application is related to commonly owned U.S. patent applicationSer. Nos. 12/166,969; 12/167,002 and 12/166,984, filed on Jul. 2, 2008,which are hereby incorporated herein by reference. This application isalso related to U.S. patent application Ser. Nos. 12/159,107 and12/159,106, filed Dec. 29, 2006, which are hereby incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to determining effectiveness ofcommunication content and to optimizing content distribution to enhancebusiness objectives, and, more particularly, to concurrently performingthese operations.

BACKGROUND

Visual information in a retail environment often takes the form ofadvertising content. Such content is inherently persuasive, and istypically designed to influence a viewer's attitudes, perceptions, andbehaviors in order to create a positive business impact, such asincreasing sales, strengthening brand awareness, or engendering consumerloyalty.

In 2002, for example, total spending on advertising content used inretail environments, commonly referred to as Point of Purchase (POP),was estimated at $17 billion in the United States and exceeded $43billion per year globally. This level of spending has garneredincreasing scrutiny among brand owner executives who are demandinggreater accountability for their marketing investments.

The need for measurable performance is increasingly urgent as well,because the average tenure of a Chief Marketing Officer has decreased toan estimated 22.9 months according to industry sources. Marketingleaders thus have precious little time to measurably demonstrate resultsfrom their marketing efforts. Marketing research, a sub-set of theresearch industry, has historically used correlational or matchedcontrol studies to evaluate advertising content performance againstobjectives. However, these “best practice” marketing researchmethodologies do not reliably reveal causation between the marketingmessage and the business result, as has been widely commented on bymarketing analysis experts (e.g., Don E. Schultz, Market ResearchDeserves Blame for Marketing's Decline, Marketing News, Feb. 15, 2005).Even so, marketing research spending is currently estimated at $8billion annually in the United States alone, which includes these typesof studies.

SUMMARY OF THE INVENTION

The present invention is directed to systems, articles, andcomputer-implemented methods for assessing effectiveness ofcommunication content and optimizing content distribution to enhancebusiness objectives. Embodiments of the present invention are directedto computer-implemented methods for a computer-implemented method,comprising conducting an experiment using experimental content todetermine effectiveness of communication content and executing, whileconducting the experiment, a machine learning routine (MLR) using MLRcontent to enhance an effectiveness metric.

Another embodiment is directed to a computer-implemented method,comprising generating a plurality of schedules each unrelated to oneanother and each comprising a plurality of time periods for presentingcontent and collecting data indicative of content effectiveness. Themethod also includes using a digital signage network comprising aplurality of geographically disparate displays and the plurality ofschedules for concurrently conducting at least two cause-and-effectexperiments on effectiveness of communication content that ensures thatexperimental content of the communication content are not confoundedusing at least two of the plurality of schedules, concurrently executingat least two machine learning routines (MLR) using MLR content toenhance a predetermined business goal using at least two of theplurality of schedules, or conducting at least one of thecause-and-effect experiments while executing at least one of the machinelearning routines using at least two of the plurality of schedules.

Another embodiment is directed to a computer-implemented method,comprising receiving a viewer visit duration (VVD) for viewers at alocation where content is to be presented, generating a schedulecomprising a plurality of time periods for implementing a machinelearning routine (MLR) based, in part on the VVD and an effectivenessmetric. Then, the MLR is executed, using a digital signage networkcomprising a plurality of geographically disparate displays, inaccordance with the schedule to determine effectiveness of the MLRcontent.

Another embodiment is directed to a computer-implemented method,comprising performing an evaluation to determine, for any given timeperiod, if using experimental content has more value than using MLRcontent for the time period. Content is then assigned to the time periodbased on the result of the evaluation.

Another embodiment is directed to a computer-implemented method,comprising receiving data gathered in accordance with a schedulecomprising a plurality of time-slot samples and executing a machinelearning routine (MLR) using content collected from within time-slotsamples to enhance an effectiveness metric.

The above summary of the present invention is not intended to describeeach embodiment or every implementation of the present invention.Advantages and attainments, together with a more complete understandingof the invention, will become apparent and appreciated by referring tothe following detailed description and claims taken in conjunction withthe accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart that illustrates processes for concurrentlyconducting cause-and-effect experiments on content effectiveness andautomatically adjusting content distribution patterns to enhanceeffectiveness metrics in accordance with embodiments of the presentinvention;

FIG. 2 is a flow chart that illustrates processes for concurrentlyconducting cause-and-effect experiments on content effectiveness andautomatically adjusting content distribution patterns to enhanceeffectiveness metrics in accordance with other embodiments of thepresent invention;

FIG. 3 shows several non-limiting examples of temporal relationshipsbetween conducting cause-and-effect experiments and optimizingeffectiveness metrics in accordance with embodiments of the invention;

FIG. 4 is a flow chart that illustrates processes for concurrentlyconducting cause-and-effect experiments on content effectiveness andautomatically adjusting content distribution patterns to enhanceeffectiveness metrics in a manner that accounts for value and urgency ofboth processes in accordance with embodiments of the present invention;

FIG. 5 is a flow chart illustrating a content-distribution andreturn-on-investment optimization methodology that uncovers correlationsof potential value to a user in accordance with embodiments of thepresent invention;

FIG. 6 illustrates processes implemented via a user interface of adigital signage network for receiving input data from a user that isused to generate an optimized schedule for presenting content on adigital signage network that maximize return on investment in accordancewith embodiments of the present invention;

FIG. 7 is a flow chart showing processes implemented via a userinterface of a digital signage network for receiving input data from auser that is used to generate an optimized schedule for presentingcontent on a digital signage network that maximize return on investmentin accordance with other embodiments of the present invention;

FIG. 8 is a flow chart showing processes for inputting data into anoptimization algorithm and how the optimization routine uses these datato generate an optimized schedule in accordance with embodiments of thepresent invention;

FIG. 9 is a flow chart showing processes for concurrently implementing acause-and-effect experiment and a machine learning routine via a digitalsignage network in accordance with embodiments of the present invention;

FIG. 10 illustrates a playlist schedule for testing the effectiveness ofdifferent content at different display locations to enhance apredetermined business goal for an illustrative deployment scenario inaccordance with embodiments of the present invention;

FIG. 11 is an illustration that shows content restrictions for thedifferent display sites and time periods of the scenario depicted inFIG. 10;

FIG. 12 is a schedule for implementing a machine learning routine whileconducting a cause-and-effect experiment for the illustrative deploymentscenario of FIGS. 10 and 11, the schedule showing “open” time periodsfor optimizing content distribution to maximize multiple business goalsin accordance with embodiments of the present invention;

FIG. 13 shows historical data collected by the content distributionoptimization routine for the scenario depicted in FIGS. 10-12;

FIG. 14 shows the expected return-on-investment for each of the timeperiods shown in the schedule of FIGS. 10-12;

FIG. 15 is a schedule for implementing a machine learning routine whileconducting a cause-and-effect experiment for the illustrative deploymentscenario of FIGS. 10-12, the schedule showing “open” time periods towhich machine learning routine content is assigned in accordance withembodiments of the present invention;

FIG. 16 shows data that was collected during the collection periods ofthe open time periods of FIG. 15;

FIG. 17 shows data for the experiment depicted in FIGS. 10-16 that areparsed out of a data stream, analyzed, and presented to the user inaccordance with embodiments of the present invention;

FIG. 18 shows data of another illustrative deployment scenario thatdemonstrates that optimization routines of the present invention cangenerate a significant improvement in return-on-investment by optimizingover content, daypart, and location;

FIG. 19 illustrates a one-day schedule with multiple business goals foranother illustrative deployment scenario in accordance with the presentinvention;

FIG. 20 is an example of a second schedule with which the user tests theeffectiveness of different content for the deployment scenario of FIG.19;

FIGS. 21-23 show representative processes for conducting a costevaluation of an experiment in accordance with embodiment of the presentinvention;

FIG. 24 illustrates representative processes for optimizing thefrequency rate of content presentation in accordance with embodiments ofthe invention;

FIGS. 25A and 25B show the results of mixing different content withinthe same time-slot sample period in accordance with embodiment of FIG.24;

FIGS. 26-28 are flow charts directed to identifying and uncoveringvaluable correlations using an automatic hypothesis generationmethodology in accordance with embodiments of the present invention;

FIG. 29 illustrates a representative embodiment of a methodology forconducting an evaluation of all displays of a digital signage networkand assigning display screen time to be under the control of either acause-and-effect experiment system or a machine learning system inaccordance with embodiments of the present invention;

FIG. 30 is a system-wide view of how the processes described in FIG. 29may be implemented for each time period on a display-by-display basis inaccordance with embodiments of the invention;

FIGS. 31-33 are flow charts that illustrate processes for continuouslyevaluating the assignment of display time periods (e.g., TSS's) toeither a cause-and-effect experiment system or a machine learning systemin accordance with embodiments of the invention;

FIG. 34A is a diagram that illustrates processes implemented by computerassistance for distributing communication content and assessingeffectiveness of such content in accordance with embodiments of thepresent invention;

FIG. 34B is a diagram that illustrates processes implemented by computerassistance for distributing communication content and assessingeffectiveness of such content in accordance with embodiments of thepresent invention;

FIG. 35 illustrates processes involving network setup and data gatheringin connection with algorithmically scheduling and presentingcommunication content consistent with constraints of a cause-and-effectexperiment in accordance with embodiments of the present invention;

FIG. 36A illustrates processes for controlling location carryovereffects in connection with distributing communication content andassessing effectiveness of such content in accordance with embodimentsof the present invention;

FIG. 36B illustrates processes for controlling location carryovereffects in connection with distributing communication content andassessing effectiveness of such content in accordance with otherembodiments of the present invention;

FIG. 37 illustrates processes for algorithmically scheduling andpresenting communication content consistent with constraints of acause-and-effect experiment in accordance with embodiments of thepresent invention;

FIG. 38A illustrates various processes involving generation of time-slotsamples in accordance with embodiments of the present invention;

FIG. 38B illustrates various processes involving assigning content totime-slot samples in accordance with embodiments of the presentinvention;

FIG. 38C illustrates an embodiment of an algorithm that may be used forparsing a schedule into time-slot samples using a complete randomizationprocess in accordance with embodiments of the present invention;

FIG. 38D illustrates an embodiment of an algorithm that may be used forparsing a schedule into sequentially generated time-slot samples inaccordance with embodiments of the present invention;

FIG. 38E illustrates processes of an algorithm that may be employed tocreate an experimental design playlist in accordance with embodiments ofthe present invention;

FIG. 38F illustrates processes of an algorithm that assigns content totime-slot samples for testing the relative effectiveness of the contentin accordance with embodiments of the present invention;

FIG. 38G illustrates processes of an algorithm that assigns content totime-slot samples using a constrained randomization process inaccordance with embodiments of the present invention, such that eachpiece of experimental content is assigned to the same number oftime-slot samples;

FIG. 38H illustrates processes of an algorithm that takes as inputsample size requirements and assigns content to time-slot samples usinga constrained randomization process in accordance with embodiments ofthe present invention to ensure sample size requirements are met;

FIG. 38I illustrates processes of an algorithm that assigns content totime-slot samples using a complete randomization process but with theaddition of optimization factor constraints in accordance withembodiments of the present invention;

FIG. 38J illustrates processes of an algorithm that assigns content totime-slot samples using a complete randomization process but with theaddition of blocking factor constraints in accordance with embodimentsof the present invention;

FIG. 39A illustrates processes of an algorithm that assigns content totime-slot samples in accordance with embodiments of the presentinvention, where the individual pieces of content are shorter than thetime-slot samples;

FIG. 39B illustrates processes of an algorithm that assigns content totime-slot samples in accordance with embodiments of the presentinvention, the algorithm ensuring that there are no location confoundsduring a duration of interest;

FIG. 40A is a block diagram of a digital signage system thatincorporates the capability for designing and deploying cause-and-effectexperiments and machine learning routines in accordance with embodimentsof the invention;

FIG. 40B illustrates a system that is configured to design, conduct, andanalyze data for cause-and-effect experiments and machine learningroutines in accordance with embodiments of the invention; and

FIG. 40C is a diagram of a digital signage network that includes variouscomponents of a DSS in accordance with embodiments of the presentinvention, including a DSN system module communicatively coupled to anexperiment system and a machine learning system; these embodimentsencompass embodiments comprising the decision tool shown in FIG. 30.

While the invention is amenable to various modifications and alternativeforms, specifics thereof have been shown by way of example in thedrawings and will be described in detail. It is to be understood,however, that the intention is not to limit the invention to theparticular embodiments described. On the contrary, the intention is tocover all modifications, equivalents, and alternatives falling withinthe scope of the invention as defined by the appended claims.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

In the following description of the illustrated embodiments, referenceis made to the accompanying drawings that form a part hereof, and inwhich is shown by way of illustration, various embodiments in which theinvention may be practiced. It is to be understood that the embodimentsmay be utilized and structural changes may be made without departingfrom the scope of the present invention.

Embodiments of the present invention are generally directed tocomputer-implemented systems and methods for assessing effectiveness ofcommunication content and optimizing content distribution to enhancebusiness objectives. Particular embodiments are directed to concurrentlyexecuting communication content effectiveness assessments, preferablyusing cause-and-effect experiments, and optimizing content distributionpatterns that maximize one or more effectiveness metrics(point-of-purchase sales, upgrades, customer loyalty, etc.) thatmaintains the validity of the cause-and-effect experiments. In generalterms, cause-and-effect experiments are controlled experiments in whichappropriate balancing, counterbalancing, blocking, randomization, andmeeting necessary sample size requirements are used to ensureconfound-free results. Effectiveness metrics refer to measured resultsof consumer behavior. Representative effectiveness metrics includesales, customer loyalty, upgrades, brand perception, and othermeasurable elements of a particular business goal.

Business goals refer to a general category of a viewer behavior thatspecifies a relationship between experiencing a piece of content andresponding to the content. Representative examples of business goalsinclude “Bar Sales,” “Room Upgrades,” and “Package Food Sales.” Otherbusiness goals include influencing attitudes and beliefs about brandsand products and/or influencing foot traffic patterns within anestablishment. A business goal is associated with at least oneeffectiveness metric (e.g., number of room upgrades), but there may be acollection of effectiveness metrics for a particular business goal(e.g., preferred customer upgrades, new customer upgrades, complimentaryupgrades). A business objective often relates to multiple business goalsand may change over time. For example, this week a user may have theobjective of maximizing both room upgrades and bar sales. Next week, theobjective may change to maximizing room upgrades while controlling barsales.

Various embodiments of the invention are directed to scheduling contentdistribution and determining, for each time period of the schedule,whether to utilize each time period for conducting a cause-and-effectexperiment or optimizing content distribution patterns that maximize oneor more effectiveness metrics in a manner that maintains the validity ofthe cause-and-effect experiment. For purposes of the present disclosure,a schedule may be a static plan or a continuously and dynamicallyupdated plan. Scheduling methodologies consistent with embodiments ofthe invention typically involve making such determinations on adisplay-by-display basis for each time period of the schedule. Incertain embodiments of the invention, time periods of a schedulecorrespond to time-slot samples as described herein and in commonlyowned U.S. patent application Ser. No. 12/166,984. Schedulingmethodologies consistent with these embodiments may further involvedynamically adjusting the schedule on a per-time period basis to achieveuser-specified requirements (e.g., desired balance of experimentationvs. content distribution optimization). According to some embodiments,systems and methods may be implemented that constantly analyze whether,for a given time period, a display should be “under the control” of acause-and-effect experiment system or under the control of a machinelearning system given the cost of using a particular time period on eachdisplay for experiments relative to a lost opportunity of using the sametime period on each display for enhancing a predetermined business goalby execution of the machine learning routine.

Systems and methods of the present invention can be implemented toexecute various types of machine learning routines to enhance oroptimize one or more effectiveness metrics. In general terms, a machinelearning routine refers to a computer-implemented methodology forlearning the relationships between actions (e.g., content), states(e.g., sign location, time-of-day, etc.) and rewards (e.g., sales,upgrades, etc.). Representative examples of useful machine learningroutines include a reinforcement learning routine, logistic regressionroutine, unsupervised learning routine, semi-supervised routine, or useof one or more neural networks. Other machine learning routines that maybe used include transduction, genetic algorithms, support vectorroutines and learning-to-learn routines, among others.

One particular machine learning methodology, reinforcement learning, hasbeen found to be particularly useful in the context of variousembodiments of the present invention. Reinforcement learning allowssystems, machines, and software agents to automatically maximizeperformance for a particular problem space. Many different algorithmscan be developed to implement a reinforcement learning routine.Reinforcement learning can be applied to problems in which there arestates, actions, and rewards. The states refer to identifiableproperties that are not under the control of the algorithm in which theproblem can exist (e.g., time-of-day, display location, weather, etc.).Actions refer to the elements that are under the control of thealgorithm (e.g., content to display). Rewards are the measurableresponses (e.g., point-of-purchase sales) to the actions generated bythe algorithm given the state when those actions were executed.

Reinforcement algorithms are designed to learn the relationship betweenactions, states, and rewards. More generally, the reinforcement learningalgorithms (and machine learning in general) learns an expected outcome(reward) that will be generated when the system is in a particular stateand a particular action is generated. Under many real world conditions,the relationship between rewards, states, and actions is probabilisticand thus the resulting reward, given a particular action and state, willvary from trial-to-trial.

For a given problem, the reinforcement algorithm is programmed to learnthe relationship between the states, actions, and rewards by testingdifferent actions in the different states and recording and analyzingthe resulting rewards. The result is a prediction of the expected rewardfor executing each action for each state. Because the system isconstantly evaluating the expected reward for an action for a givenstate, the model can learn to maximize the expected reward for a staticstate problem (i.e., a system in which the reward for an actions/statepair remains constant) and it can also adapt to a dynamic state problem(i.e., one in which the reward for a particular action/state pairchanges over time). Under some conditions, a reinforcement learningalgorithm will converge to a global optimum.

In accordance with embodiments of the invention that employreinforcement routines, the state may be defined by the time-of-day, thetype of store, and the geographical location of the business or anyother state dependent variables. For example, a particular state may bedefined by the time-of-day of the display period (e.g., 9:00-9:30 AM),the type of store (e.g., urban) and the geographical location (e.g.,Midwest). The actions relate to distributing specific pieces of contentavailable to the algorithm. The rewards are the effectiveness metrics(individual or combinations of specific effectiveness metrics) and mayinclude point-of-purchase sales, loyalty data, upgrades, etc.

A reinforcement learning routine of the present invention typicallyinvolves an explore routine and an exploit routine. It is noted thatsome implementations may use only one of these two routines or mayselect between these two routines. Exploit generally relates to showingthe content that the machine learning algorithm predicts will producethe largest reward given the current state. Explore generally relates toshowing content that is not predicted to produce the largest reward withthe goal of learning, updating, and/or verifying the expected reward ofthat content for the current state. The goal of the reinforcementroutines is to provide an understanding of the relationship between thestates, actions, and rewards.

Embodiments of the present invention are directed to systems and methodsthat facilitate user input of data associated with one or morehypotheses for a cause-and-effect experiment and data associated withone or more business goals. After entry of these and any other necessarydata, processes of the present invention are executed to ensure that,for each time period of a playlist schedule and for each display of anetwork of displays, that the system will work to maximize the utilityof the network to achieve the user's requirements indicated by theuser's input data. The user need not be further involved in theseprocesses unless involvement is desired. The user may, for example,query the system to determine the status of the network display withhigh resolution (e.g., the state of the network displays can be resolvedto a single time period of the schedule) and, if desired, implementchanges to these processes, such as by terminating an experiment orincreasing the amount to time periods allocated to explore and/orexploit routines.

Some embodiments of the invention are directed to systems and methodsfor implementing optimization of content distribution to maximizebusiness objectives exclusive of conducting communication contenteffectiveness assessments. Other embodiments involve optimization ofcontent distribution to maximize business objectives and optionallyinvoking cause-and-effect experimentation if factors indicate thedesirability of such experimentation.

Various embodiments of the present invention are directed to systems andmethods for executing machine learning routines via a digital signagenetwork. Some embodiments involve generating a playlist schedulecomprising machine learning routine (MLR) content assigned to time-slotsamples, and executing machine learning routines using the time-slotsamples. Particular embodiments involve use of reinforcement learningroutines and generating a playlist schedule comprising reinforcementlearning content assigned to time portions (e.g., time-slot samples) ofthe playlist schedule. Playlist schedules according to these embodimentsare preferably executed via a digital signage network to optimizecontent distribution patterns that maximize one or more effectivenessmetrics, such as point-of-purchase sales, upgrades, and customerloyalty, and the like.

These and other embodiments of the invention may be implemented via acomputer controllable, multiple-location content presentationinfrastructure, such as a digital signage network. It is understood thatembodiments of the invention are not limited to visual media, but mayinvolve aural, tactile, or other sensory media alone or in combinationwith visual media.

Turning now to FIG. 1, there is shown a flow chart that illustratesprocesses for concurrently conducting cause-and-effect experiments oncontent effectiveness and automatically adjusting content distributionpatterns to enhance effectiveness metrics in accordance with embodimentsof the present invention. Embodiments according to FIG. 1 involveconducting 10 a cause-and-effect experiment on effectiveness ofcommunication content that ensures that experimental content of thecommunication content are not confounded. Embodiments according to FIG.1 further involve executing 12, while conducting the experiment, amachine learning routine that distributes content to maximize apredetermined set of rewards (i.e., effectiveness metrics).

FIG. 2 shows a flow chart that illustrates processes for concurrentlyconducting cause-and-effect experiments on content effectiveness andautomatically adjusting content distribution patterns to enhancebusiness objectives in accordance with other embodiments of the presentinvention. Embodiments according to FIG. 2 involve providing 20 aschedule comprising time periods, and conducting a cause-and-effectexperiment on effectiveness of communication content using time periodsallocated for the experiment. Processes according to FIG. 2 furtherinvolve concurrently implementing a machine learning routine using MLRcontent to enhance or optimize predetermined effectiveness metrics usingtime periods of the schedule not allocated for the experiment. MLRcontent refers to the collection of content that is available forconsideration by the MLR algorithm. It may include content that isspecifically designed for the MLR, experimental content, or any othercontent.

FIG. 3 shows several non-limiting examples of temporal relationshipsbetween conducting cause-and-effect experiments and optimizing businesscontent in accordance with embodiments of the invention. Embodimentsaccording to FIG. 3 involve providing 30 a schedule comprising timeperiods to which communication content can be assigned. Processesaccording to FIG. 3 further involve conducting 32 a cause-and-effectexperiment on effectiveness of experimental content while implementing amachine learning routine using MLR content to enhance/optimize one ormore predetermined effectiveness metrics. The temporal overlap betweenimplementing cause-and-effect experiments and optimizing thedistribution of content may take several forms.

For example, and as also shown in FIG. 3, at least some of the timeperiods of the schedule that are unused for conducting one or moreexperiments may be used 34 for executing one or more machine learningroutines. According to another approach, an experiment(s) may beconducted using a first set of time periods of the schedule while asecond set of time periods of the schedule interspersed with the firstset of time periods may be used 36 for executing a machine learningroutine(s). It is understood that more than two sets of time periods maybe allocated for concurrently conducting one or a multiplicity ofexperiments and one or a multiplicity of machine learning routines.According to a further approach, for experimental content and MLRcontent that are unrelated, at least a portion of each of anexperiment(s) and a machine learning routine (s) may be implementedusing the same time period or periods.

The approaches shown in FIG. 3 and other alternative approaches can beimplemented individually or in various combinations. Also, multipleschedules may be provided and used to implement the approaches shown inFIG. 3 (and in other Figures) in a manner that ensures the integrity ofcause-and-effect experimentation while concurrently using machinelearning routines to maximize effectiveness metrics.

Traditional approaches known in the art have heretofore been unable toconduct a confound-free cause-and-effect experiment while concurrentlyexecuting a machine learning routine as depicted in FIGS. 1 and 2.Conducting a cause-and-effect experiment in the context of digitalsignage networks (DSNs), for example, generally involves implementingcarefully designed, controlled and conducted performance evaluations(experiments) that measure the impact of digitally delivered messages onspecific dependent variables (i.e., consumer behavior). These resultsgenerate insights about specific independent variables of interest(e.g., message type, message form) for particular dependent variables(e.g., metrics such as upgrades, product sales, etc.) that are ofinterest to the user. These insights are generated by carefullyscheduling content on the user's network that provide confound-freeresults to the user. Conventional approaches have been unable tofacilitate cause-and-effect experimentation while at the same timefacilitate execution of optimization routines to automatically maximizepre-specified effectiveness metrics.

In some embodiments, methods of the invention are performed using anexperiment, such as a cause-and-effect experiment as described herein.In other embodiments, methods of the invention are performed using othertypes of experiments, such as quasi-experiments or correlationaldesigns.

Previously, users have had to choose between these alternatives whenusing prior art approaches is because cause-and-effect experimentsrequires content to be distributed very precisely, such that confoundsare minimized or eliminated within the experiment. By contrast,optimization routines continuously adjust content distribution tomaximize a function (e.g., total revenue), and do so without anymechanism to ensure that confounds are not being inserted in the contentdistribution schedule. As such, conventional implementations require endusers to choose between either using a system that distributes contentfor cause-and-effect results or a system that automatically maximizesspecific effectiveness metrics.

Embodiments of the present invention are directed to systems andmethodologies that, simultaneously across a network of displays, providethe integrity of a cause-and-effect experimentation while facilitatinguse of optimization routines that maximize effectiveness metrics.Embodiments, such as that illustrated in FIG. 4, are directed toconcurrently executing 40 cause-and-effect experiments on contenteffectiveness in addition to automating content distribution patterns tomaximize consumer metrics. Embodiments of the invention may furtherinvolve balancing 42 the distribution of return-on-investment (ROI)maximization with the distribution of cause-and-effect content thattakes into account the value and the urgency of both of thesecomponents. The balancing of these components takes into account thepredicted opportunity cost for distributing cause-and-effectexperimental content during a time period instead of MLR content thatwill maximize specific effectiveness metrics. The system will selecttime periods that will maintain the integrity of the experiment (e.g.,appropriate counterbalancing and statistical assumptions necessary forthe inferential statistics) that minimizes the opportunity costsassociated with the experiment.

Opportunity cost reductions may come in many forms, including, but notlimited to, designing an experiment with unequal number of samples indifferent conditions based upon the value/priority of questions that theexperimenter is interested in answering, the cost of using specificsamples, and providing on-going cost analysis to determine the expectedcost versus benefit for finding a reliable effect given the currentresults (i.e., means and variances of the different conditions) andspecifying a stopping rule based upon statistical properties such aspower and effect size. Embodiments of the invention include optimizingthe allocation of samples to the cause-and-effect experiments and theMLR routines that minimize the cost associated with the cause-and-effectexperiments and maximize the expected reward from the MLR routines.

Because the MLR routines are learning the relationship between thestates, actions and rewards, ultimately a very rich historical databasewill exist. In addition to using this database for predicting the bestaction (i.e., content) to assign to a particular time period on aparticular digital display, this historical database can be used toprovide the user with alerts to particular relationships in the datathat the user may be interested in knowing and/or testing. Because theserelationships are not generated using experimental design, under mostconditions they will be simple correlations between independent anddependent variables. These correlation results do not provide the userwith cause-and-effect results, but simply a relationship between theindependent and dependent variables.

Because the system is learning relationships between the states and thecontent, there will be a very large set of relationships that will beavailable. One method for rapidly filtering through these possiblerelationships is to query the user about the relationships that theyfind interesting and/or valuable. By querying the user aboutrelationships that are valuable, the system can constantly be analyzingthe historical data for valuable relationships. The user can be alertedabout specific relationships in the data and given the option to run acause-and-effect experiment testing the causal effects of theindependent variables on the dependent variables, and/or, the user canindicate that the system should automatically design and run anexperiment that is valuable and meets specific criteria (e.g., predictedcost).

Those skilled in the art will readily understand that the terms“optimize,” “optimization,” “maximize,” and similar superlatives in thecontext of the present invention are not absolute terms. Rather, suchterms describe the process of either minimizing or maximizing a valuebased upon the current constraints of the problem. For example, whenoptimizing the content distribution pattern, the system constantlymonitors the performance of the actions that are available to it (i.e.,the content that can be assigned to a schedule) and assigns the contentthat, given its current knowledge, will maximize a particular value. Butbecause the optimization, or maximization routines, are working in aproblem space that is highly dynamic and the specific underlyingfunction is unknown, the system is constantly in a state of uncertaintywith respect to which action will actually minimize or maximize theobjective function. These terms are used commonly in the technicalliterature relating to machine learning. Therefore, in the context ofthe presently claimed subject matter, terms such as optimize or maximizeare synonymous with terms indicative of enhancement, betterment,improvement, increase, advancement, and the like.

FIG. 5 is a flow chart illustrating a content-distribution methodologythat may be implemented by data processing systems for continuouslyadjusting content distribution patterns for digitally distributed media,such as digital signs of a digital signage network. Embodimentsaccording to FIG. 5 involve conducting 50 experiments to generatecause-and-effect data about hypotheses entered into the system by a user(e.g., does having images of people in content impact sales?).Embodiments according to FIG. 5 further involve using 52 optimizationroutines, while conducting experiments 50, to maximize total revenueacross multiple complementary and competing categories and products.

An example of complementary products is shampoo and conditioner. Thatis, if the customer is driven to purchase shampoo, they are also morelikely to buy conditioner. An example of competing products ispre-packaged food versus in-store prepared food. If the customer isdriven to buy in-store prepared food, they will be less likely to buypre-packaged food. By using multiple effectiveness metrics (e.g.,shampoo sales, conditioner sales, prepared food sales and packaged foodsales) and specifying the relative value of these effectiveness metrics,the MLR learns maximize an objective function that takes into accountthese competing and complementary consumer behaviours.

Use of optimization routines 52 to increase or maximize the objectivefunction across multiple effectiveness metrics typically involvesoptimizing 54 the amount of time that each display of a digital displaynetwork dedicates to different content messages, and showing 55 versionsof content that are predicted to maximize the effectiveness metrics (orobjective function) for a particular screen, location, and time. Anobjective function refers to a set of relative values for each of thedifferent consumer metrics. This is used by the MLR to predict whichpiece of content will be the most effective for a particular state(display location, time-of-day, etc.). It is also used by thecause-and-effect experiment distribution system to determine theopportunity costs associated with an experiment to optimize theexperimental design and select the set of samples (time periods) thatwill minimize the overall cost of the experiment.

Embodiments according to FIG. 5 also involve analyzing 56 historicaldata collected from the optimization routines and alerting 58 the userof correlations that the user might find valuable. An example of such acorrelation is showing content with people is correlated with increasedsales in the morning, but showing content without people is correlatedwith increased sales in the evening.

FIG. 6 illustrates processes implemented via a user interface of adigital signage network for receiving input data from a user which maythen be provided to a DSN system processor. This input data is providedto an algorithm that generates a schedule for distributing content on adigital signage network that is predicted to maximize the effectivenessmetrics (or objective function). According to the embodiment shown inFIG. 6, the value of each effectiveness metric (e.g., revenue vs.guest/shopper experience) is input/provided 60 to an optimizationalgorithm processor to provide an objective function. One or morecontent distribution constraints (e.g., do not show content X on DisplayY at time Z) are input/provided 62 to the processor. Each category ofbusiness objective may have one or more effectiveness metrics. The valueof each effectiveness metric within each category of business objective(e.g., profit generated by selling different products, the value ofdifferent experimental hypotheses) is input/provided 64 to theprocessor.

A content distribution and data processing module or processor isconfigured to distribute 65 content across the network and process 67effectiveness metrics in the form of data streams (e.g., point-of-saledata). The content distribution and data processing module is configuredto continuously adjust 69 content distribution patterns in order tolearn the relationship between the content (e.g., actions) and thestates (e.g., display properties) and maximize the objective functionthat is specified by the relative values on the different effectivenessmetrics.

The following scenario represents one of many possible implementationsin accordance with embodiments of the invention. The followingrepresentative processes are implemented by a system of the presentinvention that is capable of measuring the effects of content oneffectiveness metrics. The system is preferably configured to generatethe necessary conditions for distributing the content to run acontrolled cause-and-effect experiment preferably in a manner describedhereinbelow. The following components of the experiment would be enteredinto the system by the user, as is shown in FIG. 7:

Example #1

-   -   1. What are the dependent measures of interest and what are        their characteristics?Input/provide 70 these dependent measures.        -   a. e.g., overall sales, room upgrades, bar sales, etc.    -   2. What environmental factors are of interest? Input/provide 72        these factors.        -   a. e.g., hotel size, hotel location, etc.    -   3. What content factors are of interest? Input/provide 74 these        factors.        -   a. e.g., content that differs on background color.        -   b. content that differs in tactical message (e.g., indulge            vs. efficiency).    -   4. What are the visitor visit duration (VVD) for the dependent        measures of interest? Input/provide 75 VVD.        -   a. the VVDs define the longest (or typical) period of time            that an observer can experience the digital content and the            time that they could act on the content.        -   b. the VVDs are used to define time-slot samples (TSS) that            specify independent periods of time for running a condition            of a particular study.    -   5. Experiment Urgency/Value:        -   a. specify 77 when the experiment needs to be completed.        -   b. specify 79 the value of the experimental hypotheses            (e.g., how much would the user pay to know whether the            hypothesis is true).

These dependent and independent variables (i.e., environmental andcontent factors) are used by the system to design a specific experimentin a manner as described hereinbelow. The system algorithm(s) receivethe data that is defined above and assign content to different TSSs suchthat certain factors are precisely controlled and other factors arerandomized (e.g., that which version of content is shown is randomizedacross occupancy levels). For many experiments, only a certain amount ofdisplay time is necessary for the experiment, thus leaving certain timeperiods (e.g., TSSs) “open.” For purposes of clarity, the term “open” asused in the preceding sentence refers to the present availability of agiven time period (e.g., TSS) to be used for a purpose other than forthe experiment (e.g., a machine learning routine). However, for a giventime period, a particular “business goal” may be “open.” In thiscontext, for example, if a time period (e.g., TSS) is being used toevaluate a particular business goal, such as Upgrades, another businessgoal, such as Bar Sales, is “open.”

Once the experiment has been defined, the following representativeprocesses are preferably used to fill the “open” time periods (i.e.,those not dedicated to the experiment) using a machine learningalgorithm to generate content distribution patterns that increase theuser's objective function (i.e., the values placed on different consumermetrics). Below is a description of the representative method stepsshown in FIG. 8 for inputting data into the machine learning routinesand how the routine uses this information to generate a schedule. Thisis followed by presentations of illustrative deployment scenarios thatinvolve these representative processes.

Example #2

-   -   1. Input/provide 80 effectiveness metrics for each property into        the system.        -   a. e.g., bar sales, room upgrades, restaurant sales, etc.    -   2. Input/provide 82 the value of each effectiveness metric.        -   a. Different effectiveness metrics may have different profit            margins (e.g., bar sales vs. restaurant sales) or they may            have different metrics (e.g., number of room upgrades vs.            bar sales). Defining the value of each business goal allows            the algorithm to compute a single objective function for            different pieces of content based upon these values.    -   3. Choose 83 which content to show on the network.        -   a. The user selects from their content database the content            that they want to use for the machine learning algorithm.            This content can be from a content database from previous            experiments and/or can be made explicitly for the machine            learning routine (e.g., a reinforcement learning routine            involving explore and exploit processes, as is described            below).    -   4. Specify 84 any location/time constraints on displaying        specific pieces of content.        -   a. The machine learning routine (e.g., a reinforcement            learning routine involving explore and exploit processes)            presents the content at different locations at different            times to learn which pieces of content are the most            effective. However, content may be inappropriate for some            locations or some time periods.    -   5. For each piece of content, specify 85 the VVD.        -   a. The expected (or longest) period between experiencing            (e.g., seeing) the content and acting on the content in a            measureable way.    -   6. The system schedules 86 and continuously adjusts 88 the        schedule of the times and locations in which each piece of        digital signage content is shown on each display in a digital        signage network, with the following constraints and as shown in        FIG. 9:        -   a. Identify 90 time periods or portions necessary for            completing the study.            -   i. Assign the experimental content to time periods such                that                -   1. requested experiments will be completed by the                    end of the “urgency” date (defined in Step 5.a                    above).                -   2. the cost of the experiment, with respect to the                    opportunity cost associated with running the                    experiment versus using MLR is minimized        -   b. Identify 91 time periods not dedicated to an experiment.            -   i. These periods are display periods in which there is                no experimental content.        -   c. For the time period (location, time, and sign/display),            identify 92 content (defined in Step 3 and Step 4) that can            be presented on this sign at this particular time.            -   i. Do not consider content associated that are not                appropriate for this time (e.g., advertising Happy Hour                at 4-LOAM) or on a particular sign (e.g., room upgrade                content on a sign that is not behind the check-in                counter) or a particular place (e.g., advertising a                restaurant in a hotel that does not have a restaurant).        -   d. Determine 93 whether system is “exploring” or            “exploiting” during each time period. A number of methods            for determining this include, but are not limited to the            following:            -   i. Random: Pre-determine the number of exploit periods                (e.g., 90%) versus explore periods (e.g., 10%). With                these probabilities, randomly select whether to assign                explore or exploit content.            -   ii. Semi-Intelligent: Based on historical variance,                modify the probabilities of whether to explore or                exploit. As the variance of the measurements decreases,                increase the likelihood of exploiting.            -   iii. Cost Analysis: Estimate the opportunity cost of                exploring versus exploiting, and exploit more often when                the opportunity costs are high.        -   e. If in “exploit” mode 94:            -   i. Using historical data, identify content that                maximizes the weighted business goals given the                properties of the sign and time period.                -   1. If prior knowledge does not exist, assign content                    to schedule randomly.                -   2. Use regression analysis (or other common                    predictive functions) on historical data to predict                    the best “mix” of content to maximize performance                    (the weighted business goals).                -    a. Regression analysis will use historical data                    with display, location and time variables with each                    piece of content (and combinations of content) to                    predict the best mix of content.        -   f. If in “explore” mode 95:            -   i. Identify explore content to present:                -   1. Random: From the authorized content to display,                    randomly select one piece of content.                -   2. Time Based: Select pieces of content based on the                    last time that they were explored.                -   3. Semi-Intelligently: Identify “knowledge gaps” in                    the system's historical database for the current                    display's properties (time, location, etc.) and                    present content that will fill that knowledge gap.    -   7. A content distribution and data processing module distributes        96 content across the network.    -   8. Collect and process data 97:        -   a. Generate reports for the study.        -   b. Integrate new data into historical database.    -   9. Repeat Steps 6-8 until the user terminates 98 the        explore/exploit processes.        -   a. If any information provided in Steps 1-5 changes, this            may modify the content distribution pattern in Step 6.

The following deployment scenario illustrates how the processesdescribed above can be implemented in accordance with embodiments of theinvention.

Example #3 3a. Experiment Content Distribution

Manticore is a hotel chain that owns five hotels that have a digitalsignage network. Manticore has classified their five hotels by locationtype (Urban vs. Suburban) and their size in terms of the number of rooms(Small, Medium, Large).

Manticore wants to understand whether adding a human model in anadvertisement increases the likelihood that a customer will choose toupgrade their room (Step 1 in Example #2 above). Because it costsManticore royalty fees to use a model in their advertisements, they areinterested in determining the benefit of using a model over not using amodel. They have reasons to believe that the effect of adding a modelmight be different for their Urban hotels versus their Suburban hotels(Step 2 in Example #2 above).

Manticore designed two pieces of content that are identical in all wayswith the exception of one: One piece of content has a model(RoomUpgrade-Model) while the other does not (RoomUpgrade-NoModel) (Step3 in Example #2 above). Manticore also knows that 99.9% of all customerscheck-in and make their upgrade decision within 1-hour of entering theirhotels (VVD=1 hour, TSS=2 hours; Step 4 in Example #2 above).

Using the procedures described below for designing a cause-and-effectexperiment, the system generates a schedule and assigns ExperimentalContent to the schedule. FIG. 10 shows the distribution of content forthe five Manticore hotel properties for this study. Furthermore,Manticore wants to know the answer by the end of the next day (Step 5 inExample #2 above). It is noted that, by extending the urgency date, onecan evaluate/demonstrate how the system would modulate the allocation ofdisplay time to MLR versus cause-and-effect experimentation. Using thisdata, the algorithm schedules the content to the five Manticoreproperties as shown in FIG. 10.

FIG. 10 illustrates a playlist schedule for testing the effect ofRoomUpgrade-NoModel versus RoomUpgrade-Model in Urban vs. Suburban. Inparticular, the playlist schedule shown in FIG. 10 is generated forimplementing a cause-and-effect experiment designed by Manticore Hotelsto test the performance of two pieces of content (RoomUpgrade-NoModelvs. RoomUpgrade-Model) in Urban vs. Suburban hotels. In this design, thedependent measure is the number of room upgrades, and the Hotel Size isnot a variable of interest. Therefore, the algorithm randomizes thedistribution of content over this variable. The stippled and slantedsquares indicate the time periods (e.g., time-slot sample periods)dedicated to the experiment. The time period (e.g., TSS) for roomupgrades (i.e., the dependent measure) is estimated to be 2 hours. Ascan be seen in FIG. 10, there are a number of “open” time periods thatare not dedicated to the experiment in this deployment scenario. Theseopen time periods, which may be TSSs, are shown a screened squares. Inthis case, the Urgency is to complete the study in one day (see Step 5ain Example #2 above).

3b. Demonstration Assigning Optimized Content to Open Time Slots

Manticore is also interested in using the “open” periods (shown ascreened squares in FIG. 10) to increase their ROI.

-   -   1. Manticore is interested in maximizing two business goals:        Room Upgrades and Bar Sales.    -   2. Manticore has determined that every upgrade is worth $100        while the profit margin in the bar is 40%.    -   3. Manticore has four pieces of content in their digital content        library:        -   i. RoomUpgrade-NoModel        -   ii. RoomUpgrade-Model        -   iii. Bar-Indulgence        -   iv. Bar-Efficiency    -   4. Manticore restricts the Bar content from being played in the        early morning period (4:00-8:00). Furthermore, one of their        establishments (Hotel_U_S) does not have a bar, so they do not        want the Bar content to be played at that facility. FIG. 11        illustrates these restrictions.

FIG. 11 is an illustration that shows the content restrictions for thedifferent sites and time periods. The stippled squares are the periodsof time and locations that the content cannot be played. The whitesquares indicate that the content can be played at these locations atthese particular times.

-   -   5. In this scenario, prior customer studies have found that 99%        of all customers will enter the hotel and upgrade a room within        one hour (VDD=1 hour and TSS=2 hours for embodiments that use        TSSs). By contrast, the typical time between entering the hotel        and going to the bar is 12 hours (VDD=12 hours and TSS=24 hours        for embodiments that use TSSs).    -   6. Taking this information, the algorithm now schedules the four        pieces of content for Manticore. These experimental time Periods        and the restricted periods are shown in FIG. 3, which is an        example of an explore/exploit optimization schedule. The        stippled regions indicate times that are dedicated to the        experiment. The screened squares indicate content restrictions.        The first open time period (white squares) is from 4:00 AM-6:00        AM.        -   a. It is assumed that the experiment time periods (e.g.,            TSSs) have been established.        -   b. FIG. 12 is an illustration of the open time periods            (white regions) divided into the smallest time periods            (e.g., TSS units) for the multiple business goals (two hours            for room upgrades).        -   c. For these time periods, only RoomUpgrade-NoModel and            RoomUpgrade-Model will be considered.        -   d. The explore/exploit system randomly selects between            explore versus exploit modes, with exploit mode being used            90% of the time and explore mode being used 10% of the time.            It is understood that these percentages can vary. Using a            random number generator, the system determines that this            first open time period will be an “exploit” period.        -   e. The system looks at the historical data to determine            which piece of content would be best at this time for this            facility. For Hotel_U_S, the bar content is restricted,            since there is no bar in this hotel. Using a weighted score            for the bar (see FIG. 14) and upgrades during this exploit            period, it is concluded that the best piece of content to            present is RoomUpgrade-NoModel ($1800 vs. $800 in FIG. 14).

FIG. 13 shows historical data for the number of upgrades for a given TSS(2 hours) for the two pieces of Upgrade content and for the two piecesof Bar content. FIG. 14 shows the expected ROI for each of the timeperiods. The values shown in FIG. 13 are weighted values of the ROI forupgrades ($100/upgrade) and ROI for bar sales (40% profit margin). Thebusiness goal values are specified in Step 2 of Example #3 (3b) above.The actual values in FIG. 14 are from historical data (FIG. 13) andthese business goal values. The bold outlined region under 4:00 is thetime period currently being considered. Because there is no Bar in thehotel, the data for the performance with Bar-Indulgence vs.Bar-Efficiency is actually the same.

-   -   7. Using the historical data and the business goal weighting,        the explore/exploit algorithm assigns content to all of the open        time periods and distributes the content to the displays of the        network. The results are shown in FIG. 15.

In FIG. 15, the squares with the broken diagonal pattern indicate thetimes that have been dedicated to the Room Upgrade and Bar content,respectively. The stippled squares represent “explore” time periods inwhich the algorithm randomly selected one of the lower performing piecesof content to present during that period.

-   -   8. Data is then collected from the customer. In this case, the        number of upgrades and bar sales are downloaded to the system        database. The data are then parsed to assign each piece of data        to the appropriate time period and the content that is        associated with the time period, as is shown in FIG. 16.

As can be see in FIG. 16, data is collected during the time period thatthe content was presented. The numbers in the patterned and whitesquares indicate the number of upgrades during that TSS. The barperformance for TSSs associated with Bar Sales are listed on the farright. The Hotel_U_S does not have a bar, therefore it does not have anybar sales.

-   -   a. Data for the experiment are parsed out of the data stream,        analyzed, and presented to the customer, as is shown in FIG. 17.        This is accomplished by identifying the time periods (e.g.,        time-slot samples) that are associated with the experiment, and        then analyzing by the content associated with these time        periods, such as in the manner described hereinbelow. FIG. 17        shows experimental results for the study. It can be seen that        RoomUpgrade-Model content is significantly more effective than        RoomUpgrade-NoModel content. There is no interaction between the        content type and whether the hotel is an Urban or Suburban        facility.    -   b. Historical data stored in the historical database are update        with the new data. The historical data will specify the effect        that different pieces of content (and the combination of        content) had on the business goal results (e.g., number of        updates, bar sales) as a function of the different context        variables (time, location, etc.).

Example #4

The following illustrative deployment scenario exemplifies the value ofimplementing an ROI maximization approach of the present invention. Thedata shown in FIG. 18 demonstrates that optimization routines of thepresent invention can generate a significant improvement in ROI byoptimizing over content, daypart, and location. In this representativeexample, there are four types of stores (Urban, Suburban, Exurban andRural). The system is maximizing the ROI for Morning, Afternoon, andEvening periods using three pieces of content (A, B, and C).

In this case, simply presenting the best overall content produces a2.53% increase in ROI over simply presenting each piece of contentrandomly (i.e., equally often). Choosing the best content for aparticular daypart produces a 5.11% increase over distributing thecontent equally often over the network. Choosing the content that isbest for a specific location produces a 3.53% increase in ROI overdistributing the content randomly. However, choosing the content that isthe best for all of these context variables (daypart, content, location)generates a 12.94% increase over the randomly distributed approach.

Example #5

The following representative deployment scenario illustrates additionalcomplexity that is involved when generating a playlist schedule thataccounts for multiple VVDs and multiple business goals in which amachine learning routine is running concurrently with a cause-and-effectexperiment. This example illustrates how a user can use a machinelearning routine, such as a reinforcement learning routine that employsexplore and/or exploit algorithms, to schedule content during certaintime periods of the schedule.

In this illustrative scenario, it is assumed that a digital signagenetwork is deployed in a department store, and that the network isconfigured to perform ROI measurements. It is also assumed thattime-slot samples will be used as the time periods of the playlistschedule.

One display is near the in-store bistro, where VVD has been determinedto be 45 minutes. The bistro display is running ongoing experimentsrelating to suggestions of getting a glass of wine, appetizer, desert,etc. with your meal. Additionally, experiments are run having contentrelating to merchandising for various retail departments of thedepartment store. The experimental content is interspersed with thefood-related content.

Another display is located in the book/music department of thedepartment store, where VVD has been determined to be 20 minutes. Thisdisplay runs experiments relating only to items sold in the book/musicdepartment. Another display is located near the escalators on the firstfloor, not far from, and visible from, the department store entrance.This display runs experiments relating to a variety of content,including the bistro, retail departments, and the book/music department.The overall VVD for the department store has been determined to be 70minutes. Each display has open time-slot samples of a length determinedby the algorithms described hereinbelow.

The user decides to incorporate a machine learning enhancement fordepartment store's digital signage network. For the next quarter, theuser's business goals are defined as:

-   -   (1) Increase market share for books/music. There is a competitor        store two blocks away and the user would like to take some of        their business.    -   (2) Bring more people into the bistro as an evening dining        destination, rather than just a place for a quick bite while the        customer is shopping.    -   (3) Sell more fur coats by reviving the old Lay-Away Plan,        whereby people pick out their coat and the store holds the coat        while the customer pays $150/month until the cost of the coat is        paid off.

A schedule can then be generated in a manner described herein thataccounts for the requirements and constraints described above. Thisscenario illustrates additional complexity that can be accounted forusing a playlist schedule constructed for concurrently runningcause-and-effect experiments and optimization routines in accordancewith embodiments of the invention.

Example #6

For purposes of simplicity, and to emphasize the role of multiplebusiness goals and multiple VVDs in the playlist schedule generationscenario described in Example #5 above, consider the display that islocated in the book/music department. An initial step involves definingthe experiment to answer the following question. Which will performbetter: advertisements describing book purchasing as an “investment”versus an “earned luxury” for evening versus morning shoppers? In thisillustrative example, it has been determined that VVD is 30 minutes andTSS is 60 minutes. It is assumed that a reinforcement learning routinewill be used that includes an explore/exploit algorithm.

In accordance with Steps 1-5 of Example #2 above, the following areapplicable:

The explore/exploit algorithm will use multiple business goals thatinclude:

-   -   1. Book/Music Department sales:        -   a. VVD is 30 minutes.        -   b. 15% profit margin        -   c. No time restrictions        -   d. 2 pieces of content to be used    -   2. Bistro sales:        -   a. VVD is 1 hour.        -   b. 25% profit margin        -   c. No time restrictions        -   d. 3 pieces of content to be used    -   3. Fur Coat Layaway sales:        -   a. VVD is 4 hours (often times customers will see the            layaway advertisements for the coats, leave the store, and            return to make the purchase after consulting their spouse).        -   b. 60% profit margin        -   c. No time restrictions        -   d. 3 pieces of content to be used    -   4. Step 6 of Example #2 above—Specify the Schedule:        -   a. In this case, an experiment is to run in the morning (8            AM-12 AM) and in the evening (4 PM-8 PM). The methods            described hereinbelow are used to schedule these pieces of            content as shown in FIG. 19 (screened pattern).        -   b. The explore/exploit algorithm will schedule “Bistro” and            “Layaway” content during these times.        -   c. During the afternoon period (12 PM-4 PM), the            explore/exploit algorithm will also schedule the content for            the “Book/Music” content in addition to the “Bistro” and            “Layaway” content.        -   d. The explore/exploit algorithm will choose content that is            either the best performing for that period (exploit) or will            evaluate a piece of content that was previously not the best            performer (see Steps 6d and 6e in Example #2 above).        -   e. Each piece of content will be shown for a 30 second            period and will repeat during the particular time-slot            sample.            -   i. e.g., from 8 AM-9 AM, three pieces of content will                repeat in 30 second increments:                -   1. Book/Music-Investment                -   2. Bistro-A                -   3. Layaway-A

FIG. 19 illustrates a one-day schedule with multiple business goals(Books/Music; Bistro; Layaway). The diagonal patterned squaresillustrate the TSS periods for each business goal. The patternedindicate whether the period is dedicated to an experiment, explore orexploit content. For each 1-hour period, one piece of content forBooks/Music, Bistro, and Layaway are shown. It is noted that thisschedule is constructed only for one day and one location (thebook/music department display). Often, the experiment would be run overmultiple days and potentially at multiple locations.

FIG. 20 is an example of a second schedule in which the user is testingthe effectiveness of “Bistro-A” content versus “Bistro-B” content. Inthis illustrative example, the “Book/Music” business goal is under thecontrol of the explore/exploit algorithm, and this algorithm schedulesthe content in accordance with Steps 6d and 6e of Example #2 above. Itcan be seen in Example #6 that explore/exploit time-slot samples can beconstructed with different viewer visit durations and data collectionperiods, preferably in the manner discussed hereinbelow.

According to another approach, an explore/exploit routine may use datafrom a single time-slot sample, understanding that confidence incause-and-effect is substantially lower. Importantly, however, thedescribed use of the relationship between VVD, TSS, and data collectionperiods does eliminate same-location carryover effects. To preserve theintegrity of true experimentation constraints, all explore/exploitroutine content shown during an experimental TSSs must be “unrelated.”

By way of example, a one-hour TSS may be sequentially showing 15-secondcontent clips of experimental content, exploit content (business goalA), exploit content (business goal B), and weather report content. Inthis case, data collection may be simultaneously taking place at threeindependent point-of-sale systems, one measuring the effect ofexperimental, one measuring the effect of business goal A exploiting,and one measuring the effect of business goal B exploiting. All datacollected is thus “clean” due to the unrelatedness of the communicationcontent.

According to other embodiments, two distinct and “unrelated” schedulesmay be implemented to run concurrently on the same display. Contentswitching may occur every 30 seconds or other time interval as dictatedby the two unrelated schedules. The two schedules may have verydifferent time features. For example, at least one of VVD, TSS, and datacollection periods can differ as between the two schedules. By way offurther example, each of VVD, TSS, and data collection periods candiffer as between the two schedules.

For example, both schedules can be generated to conduct cause-and-effectexperiments (e.g., true experiments). By way of further example, bothschedules can be generated to perform machine learning routines, such asexplore/exploit routines. In accordance with another example, oneschedule can be generated to conduct cause-and-effect experiments, andthe other schedule can be generated to perform machine learningroutines. It is understood that more than two schedules can beconstructed to implement a multiplicity of cause-and-effect experiments,machine learning routines, or a combination of cause-and-effectexperiments and machine learning routines.

Whenever a study is being conducted, it comes at a cost to the customer.The time periods dedicated to the study are being used to gather data orknowledge instead of being focused on generating return on investment(e.g., using an explore/exploit algorithm). The cost of the study can becalculated as the difference in revenue generated by using anoptimization algorithm versus the amount of money generated during theactual study. Because there is a measurable cost associated with runningan experiment (i.e., the opportunity cost associated with not employinga machine learning routine), embodiments of the present inventionprovide the user with the ability to automatically terminate at anappropriate or predetermined stage.

For example, the user may specify automatic termination of a study when(1) the data has demonstrated a significant result or (2) when, giventhe current effect size and the estimated variance, the cost associatedwith the study exceeds the value that the customer places on the study.Although there are known methods describing how one can determinewhether or not to continue collecting data, none of these methodsheretofore have been applied to or contemplate evaluating experimentswith digital content distribution systems.

Example #7

The following is an illustrative example of evaluating the cost ofconducting a study in accordance with embodiments of the invention. Asis shown in FIGS. 21-23, representative processes for conducting a costevaluation of a study of the present invention include the following:

-   -   1. Implement 100 a cause-and-effect experiment as described        herein.    -   2. Experiment_Value: The user specifies 102 the value of the        current experiment.    -   3. The user specifies 104 the optimization variables as        described above in Example #2.    -   4. Distribute content and collect data 106.    -   5. Analyze data 108/100:        -   a. Run 112/130 an on-going analysis of the experiment:            -   i. calculate 132 mean and standard deviations of the                content and conditions (see Step 8a of Example #2 above                and FIG. 17).            -   ii. compute 134 power analysis:                -   1. statistical power analysis specifies how many                    more samples would be needed given the current                    effect size (differences between the conditions) and                    the current variance (how much variability there is                    in the measurements) and the number of samples                    already collected.            -   iii. compute 136 the likelihood that the true effect                size is greater than an effect size that the user would                find valuable:                -   1. if the likelihood is too low (or using a power                    analysis it is too costly to find out), then suggest                    138 termination of the study.        -   b. Experiment_ROI: compute 114 the ROI generated for the            experiment up to this point (using the weights specified in            Step 2 of Example #2 above).        -   c. Explore/Exploit_ROI: given the historical data, compute            116 the predicted ROI (using the weights specified in Step 2            in Example #2 above) that could be generated by using the            explore/exploit routine versus the experiment.        -   d. Determine 118 if the value of the study exceeds the cost            of the study:            -   i. terminate or alert user 120 if:                -   Experiment_Value<(Explore/Exploit_ROI−Experiment_ROI).            -   ii. else continue 122    -   6. Present 124 estimated cost to user to determine if user is        interested in terminating study.    -   7. Repeat 126 steps 1-6 above.

The method steps of Example #7 above describe one approach for decidingwhether and when to terminate a study. Those skilled in the art willunderstand that there are other algorithms for computing when toterminate a study, and that these algorithms may be used in accordancewith embodiments of the invention.

An advantage of using a digital signage network in the context of thepresent invention is that multiple messages can be presented on a singledisplay. This provides both an opportunity and a challenge. Bothmarketing and basic memory research clearly show that humans typicallyrequire multiple presentations of a message to both remember the messageand to act upon a message. On the one hand, a digital signage networkprovides an opportunity to provide different messages over time to aviewer. However, given that it typically takes multiple presentationsfor a customer to act upon a message, if one is not careful, messagescan become ineffective when customers do not experience a message asufficient number of times to actually modify their behavior.

The challenge is that there is no prescriptive number of experiencesthat will ensure maximum benefit. The number of experiences requiredwill depend on multiple factors, including, but not limited to:

-   -   1. The strength/power of the message:        -   a. i.e., how effective the content is    -   2. The action that is required from the message:        -   a. e.g., “Buy CREST Toothpaste” vs. “Buy a HONDA Accord”    -   3. How distracted the viewer is when processing the content:        -   a. i.e., how much the viewer was able to process the content        -   b. e.g., driving in a difficult interchange vs. sitting at a            bus stop waiting for a bus    -   4. How receptive the customer is to the message:        -   a. i.e., advertising the “Calf and Steer” restaurant to a            vegetarian convention.

Embodiments of the invention involve identifying the frequency rate forpresenting content on a digital sign that automatically optimizes thepresentation frequency of content to maximize a customer's ROI. Withreference to FIGS. 24 and 25A-25B, the following example illustratesprocesses of frequency rate optimization in accordance with embodimentsof the invention:

Example #8

-   -   1. Implement 140 a cause-and-effect experiment as described        herein.    -   2. The user specifies 141 the optimization variables as        described above in Example #2.    -   3. Distribute content and collect data 142 (see Step 6 of        Example #2 above):        -   a. using the processes described in Step 6d of Example #2            above, determine whether the system will present 143 the            current best mix frequency given historical data (exploit)            or will test a different mix frequency (explore).        -   b. if in “exploit” mode, use historical data to determine            144 optimal frequency mix.            -   i. FIG. 25A shows result of historical data showing the                effectiveness of two pieces of content as a function of                the display frequency within a time-slot sample period.                FIG. 25B shows the predicted frequency mix for these two                pieces of content and a prediction that the best mix for                the time-slot sample period (that has 10 presentation                periods) is to present RoomUpgrade-NoModel 6 times and                Bar-Efficiency 4 times.        -   c. If in “Explore” mode, choose 145 a frequency mix that is            not currently the optimal frequency mix.        -   4. Analyze 146 data and update historical database (see Step            8b of Example #2 above):            -   a. store the main effect of presenting the content with                the current ratio (see FIG. 25A).            -   b. Store the combined effect of presenting the content                with the current frequency with the content that it was                paired with (see FIG. 25B).    -   5. Repeat 147 Steps 3-4 above until the user modifies Step 2        above or terminates process.

FIGS. 25A and 25B show the results of “mixing” RoomUpgrade-NoModel withRoomUpgrade-Model within a time-slot sample period. The time-slot sampleperiod can present a total of 10 presentations in this case. FIG. 25Ashows the ROI for presenting RoomUpgrade-NoModel and Bar-Efficiency 1 to10 times during the time-slot sample period. FIG. 25B shows the effectof mixing the presentations of these two pieces of content. Theoptimization algorithm returns that the optimal frequency is presenting6 samples of the RoomUpgrade-NoModel and 4 (10-6) presentations of theBar-Efficiency content for the time-slot sample period.

One of the benefits of the explore/exploit algorithm is that it isdesigned to automatically “explore” the space of content presentationpatterns to find the best, or optimal, content mix to return ROI valuefor the customer. During this exploration phase, there is a great dealof knowledge that the algorithm begins to uncover. This knowledge comesin the form of particular correlations between content (and the contentattributes, such as color, tactics, etc.), locations, customer types,time-of-day, etc. Many of these correlations are spurious correlationsthat occur due to random chance. Other correlations may have a causalcomponent to them. In order to differentiate between spuriouscorrelations and causal-effects a controlled study is needed.

The challenge is in determining which correlations to actually pursue.The explore/exploit algorithm will uncover many correlations in thedata. Some may be of significant value to the user while others will notbe of much value. Embodiments of the present invention are directed toidentifying and uncovering the valuable correlations using an automatichypothesis generation methodology (referred to herein as auto-hypothesisgeneration), a representative example of which is described below withreference to FIGS. 26-28.

Example #9

-   -   1. Implement 150 a cause-and-effect experiment as described        herein.    -   2. The user specifies 152 the optimization variables as        described above in Example #2.    -   3. The user defines 153/160 the correlation factors for the        auto-hypothesis system:        -   i. the user specifies 161 the value of different            effectiveness metrics, combination of effectiveness metrics            (objective function), which dependent measures is the user            interested in finding correlations (e.g., bar sales, room            upgrades, etc.)    -   b. the user specifies 162 factors that are of interest:        -   i. e.g., any differentiable correlations between hotel types            (e.g. Urban vs. Suburban)    -   c. the user specifies 163 meaningful effect sizes:        -   i. how large of an effect (e.g., a difference of 10%) would            trigger the system to alert the user or design a specific            experiment    -   d. the user specifies 164 whether they want to be alerted or        have the system auto-generate an experiment study:        -   i. The user can specify a maximum cost 166 that they are            willing to pay for an auto-generated experiment (see cost            evaluation of study of Example #7 above). If the predicted            cost is lower than the acceptable cost, then auto-generate            168 the experiment.    -   4. Distribute content and collect data 154 (see Step 6 of        Example #2 above).    -   5. Analyze 156/170 historical data:        -   a. compute 172 correlations between business effectiveness            metrics and the factors of interest (defined in Step 3b            above) and other factors        -   b. find 173 correlations that have effect sizes larger than            the acceptable effect sizes (see Step 3c above)        -   c. determine 174 whether the system should send alert to            user, auto-generate an experiment, or both (see Step 3d            above):            -   i. if alert, send 175 alert to user.            -   ii. if auto-generate experiment:                -   1. compute 176 expected cost of study (see Example                    #7 above) and compare to acceptable cost in Step                    3d(i) above.                -   2. If cost is acceptable, auto-generate 178                    experiment.    -   6. Repeat 159 Steps 1-5 above.

In accordance with various embodiments, systems and methods of thepresent invention may be implemented that continuously analyze alldisplays of a digital signage network to determine whether each displayshould present content for purposes of conducting a cause-and-effectexperiment or for executing a machine learning routine. Embodiments ofthe invention may be implemented to continuously analyze all DSNdisplays to effectively decide whether each display is to be under thecontrol of a cause-and-effect experiment system or under the control ofa machine learning system. This decision is preferably made by the DSNsystem based on the cost of using a particular time period on eachdisplay for experiments relative to a lost opportunity of using the sametime period on each display for optimizing a predetermined business goalby execution of the machine learning routine.

FIG. 29 illustrates a representative embodiment of a methodology forconducting an evaluation of all displays of a DSN and assigning displayscreen time to be under the control of either a cause-and-effectexperiment system or a machine learning system. The methodologyillustrated in FIG. 29 effectively hands over a period of time (e.g., aTSS) to either a cause-and-effect experiment system or a machinelearning system on a display-by-display basis, and does for each timeperiod.

As is shown in FIG. 29, a DSN system processor or module is configured(i.e., programmed to execute program instructions stored in memory) todetermine 201/203, for each time period (e.g., TSS) of each playlistschedule and associated display, if the time period is to be used by orunder the control of a machine learning routine or a cause-and-effectexperiment. If the module determines that the time period is to be usedfor a machine learning routine, appropriate content is assigned 205 tothe time period as determined by the machine learning routine. Forexample, the machine learning routine may be programmed to presentexplore content or exploit content during this time period. If themodule determines that the time period is to be used for acause-and-effect experiment, appropriate content (e.g., experimentalcontent or placebo content) is assigned 207 to the time period asdetermined by the cause-and-effect experiment. The content assigned tothe time portion is distributed and displayed 209. The processes shownin FIG. 29 are repeated for each time period for each display of thedigital display network.

FIG. 30 is a system-wide showing of how the processes described in FIG.29 may be implemented for each time period on a display-by-display basisin accordance with embodiments of the invention. A decision tool, suchas a DSN system processor or module, decides, based on system-wideconditions, whether 220 a particular time period (e.g., TSS) is to beused by or under the control of a cause-and-effect experiment or amachine learning system. If the former, the experiment systemeffectively takes control 222 of the time period. If the later, the MLRsystem effectively takes control 224 of the time period. Appropriatecontent is distributed 226 across the displays of the networkaccordingly.

Content distribution 226 is managed in the embodiment shown in FIG. 30by a server 228. The server 228 is communicatively coupled to amultiplicity of displays 229, six of which are shown in FIG. 30 forpurposes of illustration. The state of each display 229 is known at alltimes and for all time periods, shown as time-slot samples in thisembodiment. For each display 229, the current TSS and mode (experimentor MLR mode) is shown for each time, t, of the schedule that is used bythe server 228 (preferably by a DSN system processor or module) tocontrol content distribution. In some embodiments, the time between timet and time t+1 is the duration of a time-slot sample. In anotherembodiments, the time between time t and time t+1 is a duration definedby the Time Interval (TI) as described herein.

Looking vertically along the time axis, the state of each display isshown for each time increment of the schedule control (i.e., @ time t,t+1, t+2, etc.). For example, the DSN system has determined that TSS 1is to be used for executing a machine learning routine for Display 1 attime t. For time t+1, the DSN system has determined that TSS 7 is to beused for conducting a cause-and-effect experiment for Display 1.Continuing with this example, it can be seen that, for time t+2, the DSNsystem has determined that TSS 13 is to be used for conducting acause-and-effect experiment for Display 1. The state of all displays ofthe DSN system is similarly known and controlled for each time, tthrough t+n, of the schedule control in accordance with this embodimentof the invention.

FIG. 30 provides a system view that further highlights advantages andbenefits not achievable using conventional systems and techniques. Tofurther emphasize aspects of real-world implementation concernsdiscussed previously, it should be understood that every moment adisplay is being used for optimization via MLR is a moment that thedisplay is not being used for gaining insights via a cause-and-effectexperiments and vice versa. A system that would be considered optimalwith respect to allocation of display time would analyze each unit oftime for each display and determine whether that unit of display timewould add greater value by being used for cause-and-effectexperimentation versus being used to maximize business goals. Presently,no conventional system or method exists that can achieve or approachthis optimal system.

Advantageously, systems and methods of the present invention make thisoptimal allocation decision automatically by optimization algorithms ina top-level decision tool. That is, the top-level decision tool ensuresthat the value derived independently from the two subcomponents (thecause-and-effect experiment system and the MLR system) is maximizedacross the entire content distribution network, given that eachsubcomponent needs control of time periods (e.g., time-slot samples) inorder to achieve its goals.

One way in which the decision tool may reallocate control is based oninformation that accrues during and related to a cause-and-effectexperiment. The following examples are provided, which refer to timeperiods in terms of time-slot samples for illustrative purposes.

Example #10

Because there are methods for adjusting the execution ofcause-and-effect experiments while the experiments are underway, thetop-level decision tool can continuously re-evaluate the cost/benefitequation for which sub-system should have control over the varioustime-slot samples as time progresses. That is, the value of insightsgained from a cause-and-effect experiment may be overcome by the cost ofconducting/finishing an experiment during the time course in which theexperiment actually being conducted.

Based on early dependent variable data, the decision tool (or experimentsystem) can determine that the projected effect size of the factor underinvestigation is likely to be much smaller than initially expected, andas such, the experiment would take longer to conduct to reach thedesired statistical power, and therefore the cost of conducting theexperiment might exceed the expected benefit of the insight, in light ofbenefit the network owner could otherwise derive by giving control overto the MLR system.

Example #11

The value of dedicating a particular time-slot sample (on a particulardisplay, at a particular location) to the control of the experimentsystem may change as a result of the MLR system's ability to controlthat time-slot sample/display and gain more value. For example, whereasa particular time-slot sample might have been slotted to run ‘conditionx’ of an experiment, if the MLR system could gain greater value bycontrolling the time-slot sample at that display, at that location, thedecision tool could move the implementation of ‘condition x’ for theexperiment to another display on the system. That is, ‘condition x’might have been slotted to play on a display in Dubuque at 10:00 am, butnow the decision tool decides to move that condition to a differenttime-slot sample such that it plays on a display in San Diego at 10:00am instead.

As such, there are ways in which the decision tool may speed up or slowdown an experiment by shifting control of time-slot sample, and, thereare ways in which the decision tool may keep the experiment on the samepace, but re-arrange the physical location of where the experimentalconditions play in the physical world, across the network. Likewise, thedecision tool might reallocate control due to information that accruesfrom the machine learning routines.

Referring again to FIGS. 30 and 40C, and with reference to decisionprocess 220, the “experiment vs. MLR system decision tool” is a set ofalgorithms that continuously monitors the network and decides how toallocate control of each TSS for each display to the subcomponentsystems: a) experiment system 423, b) MLR system 427. The decision tool220 uses inputs from: the user regarding the value of experimentalinsights, the experiment system regarding the required samplesize/duration to meet the desired statistical power, the incomingdependent variable data as the experiment progresses, and the estimatedor known value of allowing the MLR to control the TSSs in order tomaximize current business goals.

The experiment system 423 is the subcomponent configured to receiveinputs such as dependent measures of interest and their characteristics,environmental factors of interest, content factors of interest, viewervisit duration for the dependent measure of interest, and experimenturgency/value. The experiment system 423 then generates an experiment tobe conducted on the content distribution network. The experiment system423 can estimate the expected duration/sample size required to reach adesired level of statistical power.

The MLR system 427 is a set of processor implemented machine learningalgorithms that continuously manages content distribution in order tomaximize outcomes in the data streams of interest. The MLR system 427takes inputs such as business goals and their value, the value of eacheffectiveness metric, content available to show on the network, anylocation or time constraints, and viewer visit duration associated withcontent. Content distribution 133 across the content distributionnetwork occurs under the direction of both the experiment system 423 andthe MLR system 427, (and/or the top-level decision tool 220 itself) viacomputer hardware and software (e.g., server 228/421).

FIG. 30 shows a representative network of six separate physicallocations, each having one display 229, for a total network of sixdisplays. For example, these might be six different quick serverestaurants located in six cities across the U.S. At some point in timeduring network operation, all displays 229 will be showing contentdefined by how each TSS gets populated. That is, TSSs on the network maybe 30 minutes in duration, and each piece of content may be 30 secondsin length. Thus, 60 pieces of content will play in each TSS. The contentpieces are drawn from a library of available content (e.g., stored inserver 228/421).

The mix of (up to) 60 unique pieces content that fills each 30 minuteTSS is preferably controlled either by the algorithms associated withthe experiment system 423 or the MLR system 427. However, as shown inthe diagram, any particular TSS across the network may be under thecontrol of either the experiment or the MLR algorithms (where ‘control’means choosing the content that fills the TSS and it's play order withinthe TSS). And, each TSS at any point in time may be under the control ofeither the experiment system 423 or the MLR system 427.

At the next TSS (the next 30 minute section of time), a display 229 mayshow content in a TSS that is under the control of the same subcomponentas the previous TSS, or the other subcomponent (as shown by looking downthe vertical axis of FIG. 30 as time moves forward). For example, thedecision tool 220 may decide that at TSS 1, Display 1 will show contentdefined by the MLR system 427 (to optimize business goals). Yet, whenDisplay 1 shows content in TSS 7, the decision tool 220 may calculatethat, for maximum value to the user, that TSS 7 should be under thecontrol of the experiment system 423. Thus, at any point in time,displays 229 at different locations may be under the control of thesame, or different subcomponents (experiment system 423 or MLR system427) by virtue of showing content in time-slot samples defined by (orunder the control of) either the experiment system 423 or the MLR system427.

FIGS. 31-33 are flow charts that illustrate processes for continuouslyevaluating the assignment of display time periods (e.g., TSS's) toeither a cause-and-effect experiment system or a machine learning systemin accordance with embodiments of the invention. Representativeimplementations of processes consistent with FIGS. 31-33 are describedbelow in the context of the following examples.

Example #12

-   -   1. Define 230 a cause-and-effect experiment:        -   a. define dependent and independent factors.    -   2. Input 231 urgency date and hypothesis value:        -   a. specify relative values of main effects and interactions            in experiment.    -   3. Input 232 relative values of different effectiveness metrics.        -   a. for example, each upgrade is worth $100 and every dollar            spent in the bar is worth $0.20.    -   4. Define 233 properties of required/desired experimental time        periods (e.g., time-slot samples) for cause-and-effect study:        -   a. specify all of the conditions that need to be run in the            experiment.        -   b. based upon the value of the main effects and            interactions, specify the number of samples within each            condition.    -   5. Generate 235 multiple time-slot sample scheduling assignments        that will:        -   a. complete the experiment by the urgency date.        -   b. satisfy the required experimental conditions.    -   6. Use the historical data to compute 236/241 the opportunity        cost associated with each scheduling assignment:        -   a. for each time-slot sample period in a particular            scheduling assignment generate 243 an objective value            prediction.            -   i. The objective value predictions are generated by                taking the relative values for the effectiveness metrics                and the predicted ROI from the MLR and computing 245 a                predicted objective value prediction.                -   1. e.g., For TSS 1, it is predicted that there will                    be 10 upgrades and $500 in the bar. The objective                    value for this TSS would be                    $1,100=(10×$100/upgrade)+($500×0.2).            -   ii. Sum 247 all of the predicted objective values for                the TSSs in a scheduling assignment. This value                specifies 249 the opportunity cost for NOT dedicating                the TSS to the MLR.    -   7. Choose 238 the schedule with the lowest opportunity cost.    -   8. Continuously re-evaluate 239/250 the scheduling assignment        and re-assign when a schedule with a lower opportunity cost is        identified.        -   a. Newly acquired historical data and/or new relative values            on the effectiveness metrics will change the predicted            values of different TSS. These changes can affect the least            expensive scheduling assignment.        -   b. Analyze 252 the cost for assigning enough samples to each            main effect and interaction to generate a statistically            reliable effect.            -   i. Based on the power analysis re-evaluate 253 the                remaining conditions (i.e., re-assign the sample                conditions) that need to be conducted that takes into                account the value of the individual hypotheses and the                costs associated with each hypothesis.        -   c. Identify 254 the remaining experimental conditions (e.g.,            those associated with Time-of-day, store type, etc.) that            still need to be tested.            -   i. These are the conditions that remain to be tested                that were identified in Step 4 above.        -   d. Consider 255 multiple schedule assignments that test all            of the remaining conditions.            -   i. Use the same method described in Step 6 above.        -   e. Identify 256 the schedule that has the lowest opportunity            cost.        -   f. Determine whether the opportunity cost is less than the            value of the hypothesis (Step 2 above).            -   i. If no, then alert 259 the user.

Example #13

Manticore is interested in evaluating whether a human Model in upgradecontent is more effective than No-Model on the number of upgrades.Manticore is also interested in evaluating whether having a modelinteracts with the time daypart. More specifically, Manticore isinterested in evaluating morning check-in versus evening check-in. Theprimary question of interest is whether the Model improves performanceover No-Model regardless of the time-of-day that it is shown. Thesecondary question is whether the use of a Model interacts with theTime-of-Day.

To optimize the design of the study, the user inputs the value of thetwo questions. In this illustrative example, the user specifies that theModel vs. No-Model has a value of $10,000, while the interaction has avalue of $3,000. An urgency date is set to complete the study within 30days. The user is also queried about the minimum effect size that theywould consider to be of value. The user specifies an effect size of atleast 10% would be needed to be of interest.

The system first evaluates whether the opportunity costs associated withtwo different designs are below the threshold of the value set by theuser for the answers to the two questions being asked (i.e., Model vs.No-Model and the interaction of Model with Daypart).

The first analysis is to determine the predicted opportunity cost forrunning an experiment evaluating the effectiveness of using a Model vs.No-Model. First, the system identifies the collection of time periodsthat can be used to complete the study. Using the properties of thetime-slot samples, the system uses the historical data to determine thevariance of the upgrade data and completes a power analysis to determinethe number of samples to find a significant main effect of the sizespecified by the user (10%). The power analysis predicts that 150samples are necessary (75 Model and 75 No-Model) to find a reliableeffect.

Given this collection of time-slot samples, the system uses itshistorical database to generate a prediction about the expected rewardsfor these different time-slot samples using a machine learning routine(Expected Reward for the Machine Learning Routine, denoted ER(MLR) inthis illustrative example). Also, using the historical database, thesystem makes a prediction about the expected reward for presenting theUpgrade content (Expected Reward for the True Experiment, denoted asER(TE) in this illustrative example). The predicted ER(MLR) in thisexample is $20,000 and the ER(TE) is predicted to be $15,000. Theopportunity cost is calculated as the difference between these twopredictions:OC_Model=ER(MLR)−ER(TE_Model)In this case the opportunity cost ($5,000420,000-$15,00) is below thevalue of the answer to the hypothesis related to the use of Modelswithin content ($10,000).

The system then considers the opportunity cost for answering thequestion related to the interaction (Model vs. No-Model with Morning vs.Evening). A second set of time periods are considered that only includeMorning and Evening time periods. According to the user input (describedabove), the value of the interaction is $3,000. Thus, if adding theconstraint of daypart increases the cost of the experiment by more than$3,000, then the user will be alerted to this fact. The opportunity costof generating an experiment with the interaction is calculated by:OC_Interaction=(ER(MLR)−ER(TE_Interaction))−OC_ModelThe ER(MLR) is calculated by making predictions about the MRL expectedvalue for only the Morning and Evening dayparts—other dayparts are notconsidered. In this case, the ER(MLR) is calculated to be $23,000.

Using the historical database, the predicted expected reward forpresenting the upgrade content material during these same periods is$16,000. Thus, the predicted opportunity cost for running the experimentwith the interaction is $2,000 (($23,000−$16,000)−$5,000). The initialanalysis shows that the opportunity cost of running the experiment withan interaction ($2,000) is below the value specified by the user of theanswer ($3,000).

Given that the opportunity costs for both questions are lower than thevalue to the answers, the system generates an initial schedule thatsatisfies the requirements of the study with appropriatecounterbalancing and randomization that will be completed by the 30-dayurgency period specified by the user.

As data is being collected, the system is continuously re-evaluating thepower analysis and the number of samples that are necessary to completethe study. Furthermore, the opportunity costs are also continuouslyevaluated to determine whether the predicted opportunity costsassociated with answering each question remains below the value of theanswer specified by the user.

In this example, after day 3 of the study, a power analysis of theinteraction shows that, because there is a small interaction effect, asignificant number of samples would be needed to generate a significant(and meaningful) difference (from an initial estimation of 200 samplesto 1000 samples). This increase in necessary sample size produces anopportunity cost for the interaction that exceeds the value of theinteraction specified by the user ($3000 value and a predictedopportunity cost of $12,000). That is, the effect size of theinteraction is very small. By contrast, the effect size of the MainEffect of Model vs. No-Model is very large and the predicted opportunitycost for finishing the experiment with only the Main Effect is stillbelow the value of the answer ($10,000 value and a predicted opportunitycost of $5,000). The user is alerted to the fact that the opportunitycost associated for finding a significant interaction now exceeds thevalue of the study and queried as to whether the user would like tocontinue the study with the interaction or simply run the study togenerate an answer to the interaction.

With either response, the system will need to generate a new schedule.If the user decides to continue with the study, the schedule will haveto include a significantly larger set of samples. If the user decides tocontinue with the study, but only run it with the interaction, thesystem will re-schedule the content to include content during alldayparts (not just morning and evening dayparts).

The system will continue to generate a power analysis to ensure thatthere are enough samples to complete the study with statisticalreliability and that the opportunity costs for finding an answer do notexceed the value associated with the question.

Other embodiments of this system include those in which no urgency dateis provided by the user. In this case, the system will cue up theexperiment and will constantly evaluate whether particular time periodsshould be used for this experiment or not. One method for the system toautomatically decide whether a particular time period should beallocated to the experiment is to derive a value of the time period forthe experiment versus for the MLR. The method for calculating thepredicted expected reward for the MLR is the same as that describedabove. The method for calculating the value of the time period for theexperiment is derived by taking the value of the experiment (orhypothesis being investigated) and dividing that value by the estimatednumber of time periods needed to conduct the study (using a poweranalysis). When a time period that is necessary for conducting theexperiment is being considered, the system will conduct this calculation(time periods not necessary for the experiment have zero value). If thevalue of the time period is greater than the expected reward for theMLR, then the system will allocate that time period to theexperiment—otherwise the time period is allocated to the MLR. The usercan monitor the progress of the experiment, and if the value of thehypothesis begins to increase, the user can modify the value to increasethe speed at which the system will complete the study. In anotherembodiment, the user may only specify the urgency date and no value ofthe hypothesis. Under this condition, the system will specify a scheduleand continuously update the schedule that minimizes the cost forconducting the experiment before the end of the urgency date.

Embodiments of the present invention are directed to systems and methodsthat facilitate user input of data associated with one or morehypotheses for a cause-and-effect experiment and data associated withone or more business goals. After entry of these and other necessarydata, processes of the present invention, such as those described abovewith reference to FIGS. 29-33 for example, are executed to ensure that,for each time period of a playlist schedule and for each display of anetwork of displays, that the system will work to maximize the utilityof the network to achieve the user's requirements indicated by theuser's input data. The user need not be further involved in theseprocesses unless involvement is desired. The user may, at any time,query the system to determine the state of the network displays, and maydo so at various levels of resolution—with granularity down to a timeperiod-by-time period basis (e.g., TSS-by-TSS basis) if desired. Theuser may implement changes to these processes, such as by terminating anexperiment or increasing the amount to time periods allocated to exploreand/or exploit routines, for example.

As was discussed previously, a machine learning system may beimplemented in accordance with embodiments of the present inventionexclusive of a cause-and-effect experiment system. According toembodiments of the invention, a machine learning system is preferablyimplemented using a digital signage network of a type described herein.Time periods of a playlist schedule are allocated for presenting variouscontent on each display of the DSN network in accordance with aparticular machine learning routine. Content is distributed and data iscollected in accordance with MLR algorithms for optimizing contentdistribution patterns that maximize one or more effectiveness metrics(e.g., point-of-purchase sales, upgrades, customer loyalty, etc.).Various type of MLR systems may be implemented, including thoseconfigured to execute reinforcement learning routines, logisticregression routines, unsupervised learning routines, semi-supervisedroutines, transduction routines, genetic algorithms, support vectorroutines, and learning-to-learn routines, among others, and those thatuse one or more neural networks.

According to various embodiments, an MLR can be conducted without anycause-and-effect experimentation. Under these conditions, theconstraints that a cause-and-effect experiment require can now beremoved. More particularly, when considering content to present during aparticular time period, the DSN system does not have to consider whetherthe content has the potential of confounding the cause-and-effectexperiment. Thus, when the MLR is being run without cause-and-effectexperiments, the MLR can consider all of the content that it has at itsdisposal (e.g., explore content, exploit content, etc.). Representativeprocesses for implementing content optimization routines exclusive ofcause-and-effect experiments according to embodiments of the inventioninclude those shown in blocks 92-98 of FIG. 9, for example.

Other representative embodiments of systems and methods that employ MLRroutines without a cause-and-effect experiment are those that usetime-slot samples. According to these embodiments, the system assignscontent to TSS's that are defined by the Viewer Visit Durations (VVDs)associated with the time that a customer can potentially see a displaysign and ultimately act upon the content presented by the display.Methods for defining a TSS are described herein. For a particular TSS,the MLR selects the content without considering whether the contentwould confound a cause-and-effect experiment. The MRL then executes thenecessary algorithms as shown in blocks 92-98 of FIG. 9, for example.

The following discussion is primarily directed to details forimplementing a cause-and-effect experiment in accordance withembodiments of the invention. Although primarily describingcause-and-effect experiments, many aspects of the following discussionare applicable or adaptable to implementation of machine learningsystems, such as the embodiments described hereinabove. By way ofintroduction, there are two major classes of research: experimental andnon-experimental. Embodiments of the invention that involvecause-and-effect experiments are generally directed to systems andmethods for conducting “true” experimental research and to sub-systemsand sub-processes of such systems and methods that have stand-aloneutility and usefulness. However, while systems and processes of thepresent invention described herein find particular usefulness when usedas part of a true experiment, many of the systems, processes, andmethodologies described herein find usefulness and value outside thecontext of a true experiment.

For example, various aspects (e.g., sub-systems and sub-processes) ofthe systems and processes described as part of a true experiment may beimplemented in quasi experiments, correlational studies, or other formsof non-experimental research. Implementing various system aspects andmethodologies described herein can significantly improve the efficiencyand accuracy of non-true experimental systems and methodologies. It istherefore to be understood that the processes, methodologies, systems,and devices described herein are not limited to use only within thecontext of true experimental research, but may be used advantageously inother forms of research, such as non- or quasi-experimental research andcorrelational studies.

Experiments are typically conducted to determine empirically if thereare relationships between two or more variables, and typically beginwith the formation of one or more hypotheses positing that there is arelationship between one or more independent variables and one or moredependent variables. For example, a researcher at a pharmaceuticalcompany might formulate a hypothesis that the amount of a new drug thatpatients take will be related to the blood pressure of patients. Varioustypes of experiments may be distinguished by the manner and degree towhich they are able to reduce or eliminate the effects of confoundingvariables. Confounding variables are factors that could varysystematically with the levels of the independent variable. Only “trueexperiments,” however, can empirically determine causation, which is whythe Food and Drug Administration requires that “true experiments” beused to provide data regarding the effectiveness of new drugs, forexample.

Independent variables are the variables defined or manipulated by theexperimenter during an experiment, the amount and/or frequency of a drugadministered to patients, for example. Dependent variables are thevariables posited to be predicted by the value of the independentvariable, such as the blood pressure of patients. The experimenter thenconducts an experiment to determine if there is indeed a relationshipbetween the independent and dependent variables, such as if the amountof a drug patients receive is related to the blood pressure of patientsin a pharmaceutical experiment.

Confounding variables may also influence the dependent variable. Theseconfounding variables are not of primary interest in the experiment, yetcan influence the dependent variables and therefore obscure an accuratecause and effect relationship between the independent and dependantvariables. The experimenter is trying to understand the causalrelationships between the independent and dependent variables, however,these confounding variables can render the results of an experimentuninterpretable. Some examples of confounding variables includeHawthorne effects, order effects, carryover effects such asbetween-location confounds and within-location confounds, demandcharacteristics, and/or any other factor that could vary systematicallywith the levels of the independent variables, e.g., such as the bodymass of a test subjects in the pharmaceutical experiment discussedabove.

Confounding variables make it difficult or impossible to know whichfactor (variable) caused any observed change in the dependentvariable(s). The existence of confounding variables that are notproperly controlled during the experiment renders it difficult orimpossible to make statistical inferences about causal relationshipsbetween the independent and dependent variables.

Various types of experiments may be distinguished by the manner anddegree to which they are able to reduce or eliminate the effects ofconfounding variables. The only research methodology that reliablyreveals causality is true experiments. The term “true experiment”denotes an experiment in which the following three characteristics mustexist:

1. There are at least two levels of an independent variable.

2. Samples are randomly assigned to levels of the independent variable.That is, each sample in the experiment is equally likely to be assignedto levels of the independent variable.

3. There is some method of controlling for, or eliminating, confounds.

Experiments that lack any of the above three characteristics are nottrue experiments, and are often referred to as quasi-experiments orcorrelational designs. Only true experiments allow statisticalinferences to be drawn regarding the causal relationships betweenindependent and dependent variables. Quasi-experiments and correlationaldesigns may allow relationships between independent and dependentvariables to be established, but it is not possible to determine whetherthose relationships are causal. Various types of experimental designs(including true experiments) have been described, for example, inCampbell, D. T., & Stanley, J. C., Experimental and Quasi-ExperimentalDesigns for Research, Rand McNally, (1963).

Delivering content on displays of a digital signage network withinphysical environments is rife with potential for confounds that do notexist within the Internet domain. In a physical environment, althoughpeople are generating dependent variable data (e.g., point-of sale orPOS logs, satisfaction survey responses, sensor events), it is difficultto connect the dependent variable data to the levels of the independentvariables (e.g., content on displays) to which they might have beenexposed. Consumers wander through stores and may or may not notice thedisplays or the content playing on them. Moreover, the content playedmay change while the consumer is within viewing range, thus exposingthem to multiple levels of the independent variable. Furthermore, manyother variables might influence dependent variable data, ranging frommore-or-less predictable variables, such as changing hotel occupancyrates or seasonal temperature variances, to the unpredictable, such ascompetitive marketing promotions and road construction.

Two types of confounds within the physical environment present extremelydifficult measurement-related challenges: Between-location confounds andwithin-location confounds, also referred to as between-location andwithin-location carryover effects. It is possible to have both within-and between-location carryover effects. Within-location carryovereffects occur when viewers who were present during one experimentalcondition (e.g., while control content is displayed) are still presentduring a different experimental condition (e.g., when experimentalcontent is displayed). Between-location carryover effects occur whenviewers at one location act on the content at a different location.

Embodiments of the invention relate to methods and systems that providefor determining the existence of, and measuring the strength of,cause-and-effect relationships between content being communicated andits effectiveness on recipients. Methods and systems implemented inaccordance with embodiments of the invention facilitate distribution ofcommunication content and assessment of the effectiveness of distributedcommunication content and, as discussed above, facilitate automaticoptimization of content distribution patterns to maximize return oninvestment or other pre-established business objective. Embodiments ofthe present invention provide for distribution of communication contentin a manner such that the distribution pattern enables measuring ofcontent effectiveness. Embodiments of the present invention provide forsystematic control of the pattern (i.e., timing and location) at whichcommunication content is distributed in order to control for and/oreliminate confounds.

Communication content may take many forms, including visual or aural, orany form that can impact or be detected by the human sensory system(e.g., the five senses of the human sensory system, including tactile ortouch, taste, and smell, in addition to vision and hearing).Communication content may be static, dynamic or a combination thereof.

Distributing communication content may be effected in many ways,including electronically, optically, audio broadcasting, or graphicallyor pictorially via static or dynamic images, for example. Communicationcontent may be distributed to and within a variety of physicalenvironments, including retail stores, banks, hotels, airports,roadways, railways, and other public or private spaces. Communicationcontent may be presented via stationary or mobile structures, devices,and systems.

According to embodiments of the present invention, acomputer-implemented system and method provide for generating time-slotsamples, each of which is assigned a clock time. Each time-slot samplehas a specified time duration referred to as a time-slot sampleduration, to which content may be assigned, and a data collection periodfor measuring effects of the assigned content. The data collectionperiod of a time-slot sample is a period of time during which dependentvariable data is collected. According to other embodiments, acomputer-implemented system and method provide for assigning pieces ofcontent to time-slot samples for displaying on displays for measuringeffects of the assigned content pieces.

Embodiments of the present invention provide for the distribution ofcommunication content and to assessing effectiveness of such contentconsistent with constraints of a true experiment. Embodiments of thepresent invention are directed to providing, for use in acomputer-implemented process, rules for displaying communication contentconsistent with constraints of a true experiment. The rules, which maybe time based or event driven, preferably control or eliminateconfounds, such as carryover effects. The communication content isdisplayed according to the rules. Data relating to effectiveness of thecommunication content is collected, and the effectiveness of thecommunication content is evaluated based on the collected data.

Embodiments of the present invention are directed to algorithmicallydistributing content across one or more displays such that thedistribution pattern meets the constraints of a true experiment formeasuring the effects of the content. Conducting true experiments oncommunication content distribution networks, such as digital signagenetworks or the Internet, provides for determining the existence of, andmeasuring the strength of, cause-and-effect relationships betweencommunication content and measures of business success (e.g., sales,sensor events, survey data, etc.).

Embodiments of the present invention employ algorithms to automaticallyschedule and present signage content such that the content presentationpattern precisely corresponds to the experimental design. The output ofthe algorithms may be used as the basis for parsing the dependentvariable data to correspond to the experimental conditions.

While digital signage networks, for example, present many challenges,such networks also offer ideal conditions for experiments than othermedia, such as broadcast or cable television, radio, and print. Withregard to television and radio, for example, advertisers cannot controlwhich televisions play their commercials (i.e., manipulate independentvariables), and they cannot measure the direct effect of the commercialon product sales (i.e., measure effects of the independent variable onthe dependent variable). Since most marketing research methodologieshave evolved from these media models, market researchers appear to haveoverlooked the possibility of conducting true experiments.

Digital signage networks, by way of further example, allow for precisescheduling of advertising content (i.e., the ability to preciselymanipulate independent variables). And, because displays are typicallynear the product or otherwise in an environment in which changes inbehavior can be measured, it is possible to measure behavioral changesthat arise from the content (i.e., it is possible to measure effects ofthe independent variable on the dependent variable). Also, data used toevaluate success against objectives are typically already collected in aform that can be readily used within the experiment.

According to methodologies of the present invention, the independentvariable is preferably digital signage content and the dependentvariable may be any measure with business implications (e.g., salesdata, sensor data, survey data). Using systems and methods of thepresent invention, it is possible to systematically control the pattern(i.e., timing and location) at which digital signage content isdistributed across the digital signage network in order to control forand eliminate confounds.

In the context of various embodiments of the present invention, theindependent variables correspond to the properties of the content, suchas a strategic message or even an executional element like a dominantcolor or use of a photographic image. There are always at least twolevels of the independent variable: either both are experimental contentor one level is experimental and one is control content. Experimentalcontent is the content that is hypothesized to have an impact on thedependent variable (analogues to the drug or drugs being tested in aclinical drug trial experiment). Control content is any content thatwould not be expected to impact the dependent variable (analogous to aplacebo pill in a clinical drug trial experiment). Manipulating theindependent variables involves assigning either experimental or controlcontent to be presented on signs at different times and differentlocations. The different levels of the independent variables arerandomly assigned (with constraints, as described below) to thedifferent signs and different locations. The dependent variables can beany variable that would be posited to be impacted by the content (e.g.,sales data, sensor data measuring pre-purchase behavior).

Confounding variables, as discussed above, may influence the dependentvariable and therefore obscure an accurate cause and effect relationshipbetween the independent and dependant variables. If the experiment isdouble-blind, for example, and given proper randomization, there areonly two categories of possible confounds; carryover effects (e.g.,between- and within-location confounds), which are described above, andcontent confounds.

Content confounds occur when more than one version of experimentalcontent for the same dependent variable is played during the sametime-slot during which measurement of the dependent variable is beingmeasured. Such instances render it impossible to know which contentunderlies any observed change in the dependent variable. These types ofconfounds may be eliminated by ensuring that, within a given time-slot,only experimental and/or only control content is presented.

As previously discussed, carryover effects occur when it is possible fora viewer to observe content during one time-slot corresponding to anexperimental condition and act on the content during a time-slotassociated with a different experimental condition. Again, suchinstances render it impossible to know which content underlies anyobserved change in the dependent variable. Within-location carryovereffects occur when viewers who were present during one experimentalcondition (e.g., while control content is displayed) are still presentduring a different experimental condition (e.g., when experimentalcontent is displayed). Within-location confounds may be controlled byensuring that the time-slot samples to which content can be assigned aresufficiently long to ensure that during some of the time-slot samples(e.g., half of the time-slot sample), the vast majority of the viewers(e.g., 95%) present at the viewing location were not present during theprevious time-slot sample. In this case, data are preferably onlyrecorded during the portion of the time-slot sample in which the vastmajority of viewers who would have been present during the previoustime-slot sample would have left the location.

An alternative approach involves using most or all of the data recordedduring the time-slot sample, but weighting the data more heavily towardthe end portion of the time-slot sample as compared to the beginningportion of the time-slot sample. Furthermore, any still existingwithin-location carryover effects (e.g., those that would arise from the5% or fewer consumers that would have been exposed to both versions oftest content) may be eliminated by counterbalancing the order at whichcontent is presented (e.g., ensuring that content B follows content A asoften across the experiment as content A follows content B).

Between-location carryover effects occur when viewers at one locationact on the content at a different location. Between-location carryovereffects may be eliminated by ensuring that locations within plausibletraveling distance of each other are constrained in the content theyplay such that it is not possible to leave one location while oneexperimental condition is in force and go to a nearby location and actin ways that affect the dependent variable(s) while other experimentalcontent is in force.

Two types of blocking may be employed for different reasons; blocking byoptimization factors and blocking by noise variables. Optimizationfactors are those factors at the signage location that might haveimplications for the effectiveness of the content. Such factors includesignage location, ambient lighting, socioeconomic status of viewers,dayparts, and the like. Blocking by these factors allows for factorialanalyses to measure interactions between content and optimizationfactors (e.g., measuring whether content A is more effective in themorning whereas content B is more effective in the evening). Blocking bynoise variables can be used to increase statistical power by eliminatingvariability associated with factors that impact the dependent variablethat are predictable but that are of no interest with respect to theexperiment.

Provided hereinbelow are representative examples directed todistribution of communication content and assessing the effectiveness ofsuch content in a manner consistent with constraints of a trueexperiment. These examples are provided for illustrative purposes only,and do not limit the scope or application of the disclosed principles.Rather, a wide variety of media and communication distributionarchitectures and methodologies are contemplated, including thoseinvolving print media, cellular or wireless communication devices,Internet accessed content and devices, including fixed and portable(e.g., hand-held) devices, in-store and outdoor (e.g., electronicbillboard) display systems. A wide variety of content that can becommunicated over such architectures and devices is also contemplated,including advertising content, teaching content, and way findingcontent, for example.

Although the automated experimental design methodologies describedherein are generally focused on digital signage applications, it isunderstood that such methodologies may be applied to numerous marketingcommunication tactics, including webpage design, Internet advertising,point-of-purchase printed marketing, and direct marketing, among others.For example, Internet analytics methods or web-based automatedexperimentation systems, such as the systems disclosed in U.S. Pat. Nos.6,934,748 and 7,130,808 which are incorporated herein by reference, maybe modified in accordance with the present invention to provide forimplementing true experimental design or sub-processes that haveconstraints of a true experiment.

Aspects of the present invention may be incorporated in automatedcontent distribution systems and methods that are not directed toexperimentally measuring the effects of the distributed content, butinvolve distributing content based on other constraints, such asfulfilling contract obligations. An example of such a system and methodis disclosed in U.S. Patent Publication No. 2006/0287913, which isincorporated herein by reference. In such systems and methods, contentdistribution may be performed while simultaneously measuring theeffectiveness of the distributed content in accordance with the presentinvention.

The following non-limiting examples of systems and methodologiesillustrate various embodiments of the present invention. Some of theexamples are directed to systems and algorithms that facilitatemeasuring the effectiveness of communication content consistent withconstraints of a true experiment. Some of the examples are directed tosystems and algorithms that facilitate control of the pattern at whichcommunication content is distributed in order to control for andeliminate (or significantly reduce) confounds. Some of the examples aredirected to systems and algorithms that may be implemented to facilitatenon-experimental analyses of content effectiveness, such as inquasi-experimental analyses and correlational studies.

Various embodiments of the present invention provide for automaticparsing of the dependent variable data to correspond to the experimentalconditions. FIG. 34A illustrates embodiments that involve the provision310 of rules for displaying communication content consistent withconstraints of a true experiment. In some embodiments, provision 310 ofthese rules involves creation of such rules consistent with constraintsof a true experiment. In other embodiments, previously created rules areprovided to a system that provides for displaying communication contentconsistent with constraints of a true experiment. As is further shown inFIG. 34A, the communication content is displayed 312 according to therules. Data relating to the effectiveness of the communication contentis collected 314, and the effectiveness of the communication content isevaluated 316 based on the collected data.

FIG. 34B is illustrative of embodiments directed more particularly toautomatic scheduling and presentation of digital signage content.According to FIG. 34B, a playlist and schedule for displayingcommunication content consistent with constraints of a true experimentare provided 311. A playlist refers to the order of individual pieces ofcontent, and a schedule dictates playback of pieces of content, such asthose defined by a playlist.

In some embodiments, provision 311 of the playlist and schedule involvescreation of the playlist and schedule consistent with constraints of atrue experiment. In other embodiments, a previously created playlist andschedule are provided to a system that provides for displayingcommunication content consistent with constraints of a true experiment.The communication content is distributed 313 across a digital signagesystem. The communication content is displayed 315 on displays of thedigital signage system according to the playlist and schedule. Datarelating to the effectiveness of the communication content is collected317, and the effectiveness of the communication content is evaluated 319based on the collected data.

It is to be understood that one or multiple processing devices (e.g.,PCs, mini-computers, network processors, network servers, etc.) may beused to perform one, some, or all of the processes shown in FIGS.34A-34B and in other Figures of this disclosure. For example, a firstprocessor or set of processors may be used in the creation of playlistsand schedules. A second processor or set of processors may be used todistribute content at one location or across a digital signage system. Athird processor(s) may be used to display content according to theplaylists and schedule, while a fourth processor(s) may be used tocollect data relating to content effectiveness. A fifth processor(s) maybe used to evaluate the effectiveness of content based on the collecteddata. In some embodiments, these processes and other processes discussedherein (e.g., those associated with machine learning systems) can beimplemented by one or more processors that may be networked so as toeffect communication between some or all of these processors.

In other embodiments, some or each of such processes may be implementedby processor(s) that are not networked or otherwise linked to effectcommunication therebetween. For example, a first processor(s) may beconfigured to execute a set of program instructions to implementplaylist and schedule creation, while a second processor(s) may beconfigured to execute a set of program instructions for distributingcontent to one or a number of display devices. Unless otherwiseindicated, the terms processor, computer or module (and theirvariations) as used herein and in the claims contemplate a singleprocessor, multiple processors of which some or all may becommunicatively coupled, disparate processors (single of sub-networks)that are not communicatively coupled together, and other configurationsof processing resources.

FIGS. 35 and 36A-36B illustrate processes related to algorithmicallyscheduling and presenting communication content consistent withconstraints of a true experiment in accordance with embodiments of thepresent invention. FIG. 35 shows various processes involving networksetup and data gathering in connection with algorithmically schedulingand presenting communication content in accordance with embodiments ofthe present invention.

According to the illustrative example shown in FIG. 35, setting up thedigital signage network setup involves determining display locationsthat facilitate control, reduction, or elimination of confounds, such ascarryover effects. For example, setting up the network may involvedetermining locations 330 in which at least a predetermined percentage(e.g., 95%) of customers would not have visited another locationdisplaying experimental or control content. It is not critical that avalue of 95% is chosen. However, it is understood that the greater thevalue chosen, the less likely it is that the result could underestimatethe precise amount of the return on investment from the content. Thevalue of 95% is simply large enough that, with proper counterbalancing,the impact of carryover effects would be almost nonexistent.

It is important to ensure that the vast majority of viewers will nothave an opportunity to see the message at one site and act upon it atanother site that is playing different control or experimental content.Instances of this happening would be instances of carryover effects,which can confound the results of the experiment. For example, if onewere conducting experiments on displays in automobile dealerships, onewould need to know which dealerships are close enough in proximity toeach other such that a viewer could see content in one dealership andpurchase vehicle in another dealership partaking in the experiment. Thiscan be accomplished as part of the digital signage network setup. Forexample, the software could prompt the installer to select all of thelocations across the network at which viewers could plausibly visitafter leaving their dealership (e.g., other dealerships in the samegeographic region).

Network attributes and optimization factors present at sign locationsare preferably identified 332 at part of the digital signage networksetup. Such factors may include characteristics of each site thatpredictably impact the value of the dependent variables at the locations(e.g., store size, socioeconomic class, other advertising efforts,daypart differences in the typical number of viewers at the location).These factors then become blocking factors in the experiment.

There are two categories of blocking factors. One category includesthose factors in which the experiment would test for interactions, andthat would have implications for strategic decisions about what contentto display (e.g., content A might be more effective at lowSocio-Economic Status (SES) dealerships whereas content B might be moreeffective at high SES dealership). The other category of blockingfactors are those that do not have obvious implications for whichcontent to show, but that should nonetheless be blocked against in orderto increase the statistical power of the experiment. Again, thesefactors can be specified during the software installation process andupdated thereafter.

Network setup also includes estimating sample size requirements for theexperiment 334. Ideally, a statistical power analysis is preferably usedto calculate how much data is needed to find statistically significantresults of at least some minimum magnitude that is of business interest.

Control and experimental content are defined 336 as part of the networksetup. Control content (i.e., the placebo) can be any message that isneither intended nor likely to influence the desired behavior, such aslocal weather or news, or messages about a product or service that isunrelated to the dependent variable. Experimental content is the contentthat is hypothesized to cause a change in the dependent variable. It isnoted that, under some circumstances, experimental content for onehypothesis can serve as control content for a different hypothesis.

Data regarding the maximum duration that the vast majority of viewersspend at the site conducting their business is acquired 338 and used tocontrol for carryover effects. Examples of processes for eliminating orcontrolling for location carryover effects are described in commonlyowned U.S. patent application Ser. No. 12/166,984, filed on Jul. 2,2008, which is incorporated herein by reference.

FIG. 36A illustrates processes for controlling (e.g., reducing oreliminating) location carryover effects in connection with distributingcommunication content and assessing effectiveness of such content inaccordance with embodiments of the present invention. FIG. 36Aillustrates how within-location carryover effects are controlled if themaximum duration at which 95% of customers would spend at the signagelocation is 30 minutes. In this illustrative example, the time-slotsample 322, 324 during which content is played is double the maximumduration at which 95% of customers spend at the location.

Data recording does not begin until 95% of the customers who werepresent during the previous time-slot sample would have left the signagelocation. In this example, data are only recorded during the last 30minute portion 323, 325 of the time-slot sample 322, 324. It is notedthat the time interval for each location is preferably represented bythe smallest unit of time across which dependent variable data can bemeasured. For example, sales data collected in some point-of-salesystems is provided in units of seconds, whereas other systems reportsales only across units of hours. FIG. 36B illustrates processes forcontrolling location carryover effects in connection with distributingcommunication content and assessing effectiveness of such content inaccordance with other embodiments of the present invention.

FIG. 37 illustrates processes for algorithmically scheduling andpresenting communication content consistent with constraints of a trueexperiment in accordance with embodiments of the present invention. Theprocesses shown in FIG. 37 illustrate various actions of an experimentaldesign and execution process of the present invention. FIG. 37 isintended to illustrate a comprehensive system that incorporates numerousfeatures that facilitate scheduling and presenting communication contentconsistent with constraints of a true experiment. It is understood thatall of the features shown in FIG. 37 need not be incorporated in asystem and methodology of the present invention. Selected feature(s)shown in FIG. 37 may be utilized in stand-alone applications or combinedwith other features to provide useful systems and methods in accordancewith embodiments of the invention. FIGS. 38A-39B, for example,illustrate various useful combinations of the features shown in FIG. 7.Many combinations of the features shown in FIG. 37 may be implemented innon-experimental systems, such as quasi-experimental systems and thosethat employ correlational or regression analyses or artificial neuralnetworks.

Many of the processes shown in FIGS. 37-39B (and figures associated withvarious machine learning embodiments discussed above) have inputs thatare typically received from other processes, systems (e.g., POSsystems), sensors (e.g., presence sensors), or from a user, amongothers. These inputs include the following: duration data for each pieceof content that is being tested for effectiveness (CD); duration ofinterest (DI) after which the content is viewed not to be of interest ifthe content caused a change in the behavioral or transactional databeing measured; pair-wise content relatedness data (CR) (i.e., iscontent A expected to differentially impact the same behavioral ortransactional data as content B?); pair-wise location relatedness (LR)(i.e., the likelihood that viewers can be exposed to content at locationA and behave at location B within the above stated duration ofinterest); optimization factors present at sign location (OF); estimatedsample-size requirements, which may be optional, for how many time-slotsamples are required for each piece of content, by optimization factors(SS); maximum duration that a certain percentage of target viewers(e.g., 95%) spend at the sites where displays are located (viewer visitduration or VVD); time intervals (TI) for data collection/aggregationfor data streams of interest that target viewers can affect during visitto the site (TI); blocking factors (i.e., the most powerful factors thatare predictive of dependent variable data but that are not per se ofinterest for optimizing content); absolute placebo content; andexperimental content.

Viewer visit duration is an important parameter that represents themaximum time that a specified percentage of viewers spend at a location.VVD is typically calculated from individual VVDs for many viewers,understanding that there will be a distribution of individual VVDsdepending on a large number of factors that influence the time anindividual spends at a location. Precision of VVD depends on the size ofthe location. A small location, e.g., a small store, would havewell-defined opportunities for seeing the content and then acting on thecontent within a few minutes.

Viewer visit duration may be determined in a variety of ways, such as byestimating the VVD based on an expected VVD. Determining the VVD may bebased on one or more factors, including, for example, transactionaldata, prior sales data, sensor data (e.g., proximity or presence sensordata), and observational data. Other approaches for determining the VVDare discussed in illustrative examples provided hereinbelow.

It is understood that some “viewers” will never see (or comprehend)displayed content, but may nonetheless purchase an advertised item(generalized to the behavior being measured). Other viewers will see thecontent and not buy, and other viewers will both see and buy anadvertised item. In this regard, methods of the invention are directedto revealing the difference between measured behavior as a function ofcontent (experimental vs. control) being displayed. It is noted thatthis behavior difference being measured will also be a function ofdisplay location (e.g., in an obscure corner where few will see it vs. avery conspicuous position where all will see it). If the display is seenby few/none, then the most compelling content (FREE Flat Screen TVsToday!!!) will result in virtually no difference for measured behavior(picking up the free TVs).

Location is an important term that refers to the physical space withinwhich the viewer can be both exposed to levels of independent variables(e.g., in the form of digital signage content) and cause a change independent variable data (often dependent variable data will consist ofpoint-of-sale or sensor data) corresponding to the independentvariables. Often, the location in a retail environment is the physicalspace owned by the retailer. However, there are some circumstances whenthe location will be a subset of the space owned by the retailer. Forexample, consider the case of a hotel lobby having a display nearby thecheck-in desk, where an experiment is testing the relative effectivenessof two pieces of digital signage content designed to increase theprobability that guests will upgrade to a nonstandard room. In thiscase, the location would be the hotel lobby area (and not the entirehotel) because viewers could only be exposed to the content within thehotel lobby, and it is very unlikely that viewers would upgrade to anonstandard room other than during their first visit to the hotel lobby.As such, this is a controlled physical space allowing for precise VVDs.

In the case of a city having a single outdoor display and multipleretail establishments where consumer behavior is measured (e.g., bypurchasing an advertised product presented on the city's single outdoordisplay), for example, VVD becomes much less precise. Shopping mallenvironments typically fall somewhere between a controlled locationallowing for precise VVDs and the exemplary city scenario discussedabove. By way of contrast, it is noted that the most controlledsituation is a location represented by a person sitting at a computerdisplay, responding to (i.e., behaviorally acting on) content by way ofmouse clicks and/or keystrokes.

As was discussed previously, carryover effects occur when the effects ofone level of an independent variable persist when attempting to measurethe effects of another level of the same independent variable. Thesolution to controlling for or eliminating carryover effects provided byembodiments of the present invention is to ensure that sufficient timehas passed between (1) changing levels of independent variables; and (2)collecting data after changing levels of an independent variable.

One way to ensure that carryover effects are eliminated in the contextof digital signage content is to wait very long periods between changesof levels of independent variables and/or wait very long periods betweenchanging levels of an independent variable and collecting dependentvariable data. For example, one could only show one level of anindependent for a week or more at a time. Then, by collecting dataduring the entire week, it would be unlikely that many of the datapoints collected during the week would be impacted by a different levelof the independent variable (e.g., “save money” in this example).However, such an approach severely limits the number of instances acrosstime that levels of independent variables can be changed.

Those skilled in the art will appreciate that the speed with whichconclusions can be generated from experiments is directly related to thenumber of instances across time that independent variables can bemanipulated. Embodiments of the present invention advantageously providefor use of VVD and TI as inputs to determine how often changes in thelevels of an independent variable occur, thus allowing one to controlfor or eliminate carryover effects while changing independent variablelevels as frequently as possible.

Referring again to FIG. 37, a schedule is parsed 340 into time-slotsamples. Parsing the schedule is essential for eliminating carryovereffects. Parsing typically involves algorithmically parsing the schedulesuch that time-slot samples can be assigned to the schedule or scheduleswhich dictate playback of the content.

Creation 342 of a playlist involves algorithmically assigning content totime-slot samples such that the content distribution pattern (i.e.,timing and location at which content is played) meets the constraints ofthe experiment. This may be accomplished, for example, by ensuringexperimental and control content is not confounded 345, randomlyassigning content to time-slot samples with specific constraints thatensure blocking 346 by network optimization factors (i.e., factors thatare being studied), blocked 347 by other factors that can be controlledand predicted but that are otherwise not of interest in the study (i.e.,noise factors), counterbalancing 348 for order effects, randomizing 349across uncontrolled factors, ensuring that the design is balanced 350such that there is roughly an equal number of time-slot samples acrossblocks, and meeting 344 established sample size requirements.

The content is distributed 352 according to the playlist schedule.Ideally, this process 352 and associated algorithms are embedded withincontent management software, so that the content can be automaticallydistributed according to the created playlist schedule. A report of thealgorithmic processes discussed above is preferably generated 354. Thereport preferably identifies what content was presented, and when andwhere the content was presented. The report may also indicate whatdependent variable data to code, and any optimization, noise, andblocking factors were present or used. Other data pertinent to processesor performance impacting the algorithms may also be included on thegenerated report. It is understood that these and other data/informationis recorded so that a report of the algorithmic processes may begenerated. The report preferably specifies which dependent variable tocode within each time-slot sample, and which dependent variable data touse or discard due to possible contamination by carryover effects orother confounds.

Dependent variable measures are parsed 355 by experimental condition,which may use data of the generated report. For example, dependentvariable data (e.g., POS sales data) is preferably time and locationstamped, such that this data can be automatically parsed according tothe experimental conditions for analysis (and/or for use by a machinelearning system, such as those described hereinabove).

FIG. 38A illustrates various processes involving generation of time-slotsamples in accordance with embodiments of the present invention.According to FIG. 38A, viewer visit duration that target viewersnormally spend at a site where displays are located is received 353.Time intervals for data collection or aggregation for data streams ofinterest that target viewers can affect during their visit to the sitesare received 357. Using viewer visit duration and the time intervals, anumber of time-slot samples needed to measure effects of contentassigned to the time-slot samples are determined 359, and a datacollection period associated with each of the time-slot samples isdetermined.

Embodiments of the present invention, as exemplified by the processesshown in FIG. 38A, generate “samples,” referred to herein as time-slotsamples, to which content can be assigned for measuring the effects ofthe assigned content. These “samples,” and the methodologies thatgenerate such samples, have significant value and represent an endproduct that can be utilized by a purchaser of these samples to test theeffectiveness of content.

FIG. 38B illustrates various processes involving assigning content totime-slot samples in accordance with embodiments of the presentinvention. According to FIG. 38B, content relatedness data thatidentifies each piece of content as an experimental content piece or acontrol content piece relative to other pieces of content is received361. The processes of FIG. 38B further involve algorithmically assigning363 the experimental or control content pieces to time-slot samplesusing the content relatedness data. The content pieces assigned to aparticular time-slot sample exclude non-identical experimental contentpieces relative to an experimental content piece previously assigned tothe particular time-slot sample.

The processes shown in FIG. 38B, in one sense, describe a technique ortool (e.g., software) that can be used to increase the speed andaccuracy of conducting experiments on the effectiveness of content. Atechnique or tool implemented in accordance with FIG. 38B represent avaluable end product that provides utility to one that wishes to conductexperiments on the effectiveness of content. By way of analogy, and inthe context of the biological research domain, tools are developed andused to increase the speed and accuracy of conducting experiments on,for example, cancer cells and for decreasing the cost of conducting suchexperiments. For example, genetic sequencing tools have been developedto automatically control the steps of genetic sequencing. In a similarfashion, tools and techniques implemented in accordance with FIG. 38Bmay be used to increase the speed and accuracy of conducting experimentson the effectiveness of content, and to decrease the cost of conductingsuch experiments.

FIG. 38C illustrates an embodiment of an algorithm that may be used forparsing a schedule into time-slot samples using a complete randomizationprocess in accordance with embodiments of the present invention.According to FIG. 38C, the duration of time intervals (TI) for eachdisplay location is identified and quantified 362. The viewer visitduration (VVD) for each location is determined 364. As discussedpreviously, a TI represents the smallest unit of time across whichdependent variable data can be measured, and VVD is the maximum amountof time that a predetermined percentage (e.g., 95%) of the viewers spendat the location during any one visit.

Time-slot sample duration (TSSD) is determined 366 for each displaylocation. Time-slot sample duration is a specific duration of time thata time-slot sample lasts. During a TSSD, different experimental andcontrol content is played, preferably in a manner such that there is nooverlap that would produce confounds. According to one approach, and asindicated at blocks 368, 370, and 372 of FIG. 38C, time-slot sampleduration may be computed as follows:Is TI≧VVDIf No, then TSSD=VVD*2If Yes, then TSSD=TI+VVD  [1]

It is noted that if the TI is not equal to nor greater than the VVD(e.g., TI is 1 second) in Formula [1] above, then half of the durationof the time-slot sample duration will include viewers that were notpresent for content from the previous time-slot sample. Importantly,only data collected during this second half (i.e., the data collectionperiod of the TSSD in this example) is included in the analysis, and inconjunction with counterbalancing, this eliminates carryover effects.

A randomization process ensues, by which time intervals are subject torandom selection 376. The algorithm randomly selects any “open” timeinterval that begins at least one of a particular location's TSSDs afterthat location's opening time. The term “open” time interval in thiscontext refers to a time interval that does not already have a time-slotsample associated with it.

A time-slot sample is assigned 377 to begin one TSSD of that locationprior to the beginning of the randomly selected TI. This process 376,377, 378 continues until no legal TIs remain to assign a TSS. It isnoted that time-slot samples are selected with the following constraint:time-slot samples subsumed by previously selected time-slot samples areexcluded (e.g., if content is already being played from 9:01-9:20, thesystem does not choose 9:01-9:20 for candidate slots).

FIG. 38D illustrates an embodiment of an algorithm that may be used forparsing a schedule into sequentially generated time-slot samples inaccordance with embodiments of the present invention. Processes 362-372of FIG. 38D are the same as the corresponding processes of FIG. 38C.Processes 376, 377, and 378 of FIG. 38C are illustrative of a completerandom time-slot sample generation methodology. Processes 383, 373, 375,379, and 381 of FIG. 38D are illustrative of a sequential time-slotsample generation methodology.

According to the sequential time-slot sample generation methodology ofFIG. 38D, creating time-slot samples for each location 374 involvesselecting 383 a location at which content is to be presented. Thebeginning of the first TI that is TSSD from the location's opening timeis found 373. A TSS is assigned 375 to begin one TSSD before thebeginning of the TI. This process 373, 375 is repeated 379 for theclosest TI which is TSSD away from the end of the previous TSSD untilthe closing time is reached 381. This TSS creation process 374 isrepeated for each selected location 383. Generating time-slot samples ina sequential manner as shown in FIG. 38D generally results in achievinggreater efficiency of TI utilization.

Another means of quickly generating results needed to evaluate contenteffectiveness is the ability to use multiple locations on a network,each having a display capable of showing content. Each location can beproducing time-slot samples needed to fulfill the quantity of data tovalidate a hypothesis. In general, the rate of data generation scaleswith the number of locations, e.g., ten locations working in concert cangenerate about ten times the samples as a single location. Thisarrangement leads to the added possibility of learning aboutinteractions between content effectiveness and display locations.

The methodology disclosed in this application also allows the ability tosimultaneously test multiple independent variables during the sametime-slot samples, providing that the content associated with thedifferent independent variables is unrelated. This is becauseexperimental content for one independent variable can be control contentfor another independent variable. Using this technique further increasesthe speed of experimentation as it is possible to simultaneously conductexperiments addressing multiple business objectives, thus liberatingdisplay time to achieve business goals.

FIG. 38E illustrates processes of an algorithm that may be employed tocreate an experimental design playlist in accordance with embodiments ofthe present invention. The algorithm shown in FIG. 38E involves ensuringthat experimental and control content is not confounded 382. Accordingto the approach illustrated in FIG. 38E, each piece of experimentalcontent is randomly assigned to a time-slot sample. This process ensuresthat two pieces of content that are being compared with one another withrespect to impact on the dependent variable are never played within thesame time-slot sample.

The process of random assignment is repeated with the constraint thatonly control content is assigned to the same time-slot sample as anypiece of experimental content. This ensures that there are no locationconfounds. It is noted that it is valid to assign experimental contentfor one hypothesis to a time-slot sample that already containsexperimental content for another hypothesis, provided that the contentcan serve as a control for the experimental content for the otherhypothesis. That is, one can run two experiments at once provided thatthe hypotheses are orthogonal (and can additionally run one or moremachine learning routines as described hereinabove).

The algorithm of FIG. 38E may further involve blocking by optimizationfactors 387. This allows for factorial analyses to measure interactionsbetween content and optimization factors. The algorithm shown in FIG.38E may also involve blocking by noise factors 388 in order to increasestatistical power. These processes preferably continue to assign contentto time-slot samples until main effect and interaction effect samplesize requirements are satisfied and the design is balanced. Thealgorithm may further provide for counterbalancing 389 for ordereffects. Within each time-slot sample, the order in which individualpieces of content are displayed is counterbalanced using knowntechniques (e.g., Latin Squaring).

FIG. 38F illustrates processes of an algorithm that assigns content totime-slot samples for testing the relative effectiveness of the contentin accordance with embodiments of the present invention. The algorithmshown in FIG. 38F involves selecting 502 any time-slot sample betweenthe experiment's starting and ending points that has not already beenassigned experimental content. The algorithm further involves randomlyselecting 504 any piece of experimental content and assigning 506 theselected experimental content to play during the entire duration of theselected TSS.

The processes shown in blocks 502, 504, and 506 are repeated 508 untilall time-slot samples under the control of the Experiment System arefilled with experimental content. A report of the algorithm's output maybe generated 510. The report may contain various information, such asthat previously described with reference to FIG. 37. It is noted that ifthe time-slot samples are tagged with attributes, this will allow forhypotheses to be generated based on any interactions that are foundbetween the content assigned to time-slot samples and the attributes ofthe time-slot samples and enable exploratory data analysis.

Under many experimental circumstances, it is desirable to have eachlevel of the independent variable (or variables) assigned to the samenumber of samples. FIG. 38G illustrates processes of an algorithm thatassigns content to time-slot samples using a constrained randomizationprocess in accordance with embodiments of the present invention suchthat each piece of experimental content is assigned to the same numberof time-slot samples. The algorithm shown in FIG. 38G involves selecting520 any time-slot sample between the experiment's staring and endingpoints that has not already been assigned experimental content. Thealgorithm further involves randomly selecting 522 any piece ofexperimental content and assigning 524 the selected experimental contentto the selected TSS.

The processes shown in blocks 520, 522, and 524 are repeated 526 withthe constraint that each piece of experimental content is assigned 526to the same number of time-share samples. A report of the algorithm'soutput may be generated 528, as discussed previously.

Under some experimental circumstances, the experiment might have beendesigned manually or using off-the-shelf statistical software, or using,for example, an expert system as described hereinbelow, in which casethe sample size requirements for various experimental and controlcontent would have been specified. FIG. 38H illustrates processes of analgorithm that takes as input such sample size requirements and assignscontent to time-slot samples using a constrained randomization processin accordance with embodiments of the present invention to ensure samplesize requirements are met. The algorithm shown in FIG. 38H involvesrandomly selecting 540 the number of time-slot samples required for allcontent samples. The algorithm further involves randomly assigning 542experimental content to the selected content samples. It is noted thatthe remaining time-slot samples that were not required because samplesize requirements have been met may be filled with content that isoptimized for business results, rather than for testing any hypothesis.

FIG. 38I illustrates processes of an algorithm that assigns content totime-slot samples using a complete randomization process but with theaddition of optimization factor constraints in accordance withembodiments of the present invention. The optimization factor constraintcan be added to the equal sample size or to the predeterminedsample-size processes in an analogous fashion. It is noted that eachcontent sample would preferably have metadata identifying theoptimization factors with which it is associated, and the time-slotsamples would also have metadata identifying which optimization factorsare associated with the time-slot sample.

The algorithm shown in FIG. 38I involves randomly selecting 550 any(first) piece of experimental content, and randomly selecting 552 any(first) time-slot sample between experiment starting and ending points.The randomly selected (first) piece of experimental content is assigned554 to the selected (first) time-slot sample.

The algorithm of FIG. 38I involves randomly selecting 556 another(second) time-slot sample with the constraint that it has a differentlevel of optimization factor than a previously selected (first)time-slot sample. The selected (first) piece of experimental content isassigned 558 to this (second) selected time-slot sample. Theabove-described TSS selection processes are repeated 560 until theselected (first) piece of content has been assigned to one TSS in alllevels of the optimization factor.

The algorithm of FIG. 38I further involves randomly selecting 562 any(second) piece of experimental content, and repeating 564 processes552-560 for this next (second) piece of experimental content. Theprocesses of blocks 550-564 are repeated 566 until the maximum number oftime-slot samples have been filled without resulting in an unbalanceddesign (i.e., until there are fewer time-slot samples than the number ofoptimization factors multiplied by the number of pieces of experimentalcontent.

FIG. 38J illustrates processes of an algorithm that assigns content totime-slot samples using a complete randomization process but with theaddition of blocking factor constraints in accordance with embodimentsof the present invention. The blocking factor constraint can be added tothe equal sample size or to the predetermined sample-size processes inan analogous fashion. It is noted that each content sample wouldpreferably have metadata identifying the blocking factors with which itis associated, and the time-slot samples would also have metadataidentifying which blocking factors are associated with the time-slotsample.

The algorithm shown in FIG. 38J involves randomly selecting 602 any(first) piece of experimental content, and randomly selecting 604 any(first) time-slot sample between experiment starting and ending points.The randomly selected (first) piece of experimental content is assigned606 to the selected (first) time-slot sample.

The algorithm of FIG. 38J involves randomly selecting 608 another(second) time-slot sample with the constraint that it has a differentlevel of blocking factor than a previously selected (first) time-slotsample. The selected (first) piece of experimental content is assigned610 to this (second) selected time-slot sample. The above-described TSSselection processes are repeated 612 until the selected (first) piece ofcontent has been assigned to one TSS in all levels of the blockingfactor.

The algorithm of FIG. 38J further involves randomly selecting 614 any(second) piece of experimental content, and repeating 616 processes604-612 for this next (second) piece of experimental content. Theprocesses of blocks 602-616 are repeated 618 until the maximum number oftime-slot samples have been filled without resulting in an unbalanceddesign (i.e., until there are fewer time-slot samples than the number ofblocking factors multiplied by the number of pieces of experimentalcontent).

FIG. 39A illustrates processes of an algorithm that assigns content totime-slot samples in accordance with embodiments of the presentinvention. The embodiment shown in FIG. 39A is directed to algorithmthat assigns content to time-slot samples where the individual pieces ofcontent are shorter than the time-slot samples. The algorithm of FIG.39A ensures that there are no content confounds and allows the sametime-slot samples to be used to test multiple hypotheses (i.e., allowsunrelated independent variables to be tested within the same time-slotsamples). This is analogous to being able to test multiple drugs on thesame patients, which saves time and money. For example, in a drugtesting scenario, one can test a topical analgesic cream on the samepatient who is being used to test a halitosis cure. That is, the topicalanalgesic cream should not impact halitosis and the halitosis cureshould not impact a skin condition. However, one would not want to testa treatment for halitosis on the same patients who are being used fortesting a new toothpaste, for example.

The algorithm shown in FIG. 39A involves randomly selecting 640 any opentime-slot sample between experiment starting and ending points. A pieceof experimental content is randomly selected 642, and the selected pieceof experimental content is assigned 644 to the selected TSS. Thealgorithm of FIG. 39 further involves randomly selecting 646 a piece ofexperimental content with the constrain that it is unrelated to contentalready assigned to the TSS. The selected piece of experimental contentis assigned 648 to the selected TSS. The processes of blocks 646 and 648are repeated until it is not possible to add a piece of experimentalcontent without having the sum of the durations of all of the selectedexperimental content exceed the duration of the TSS or until there areno unrelated experimental content pieces remaining, whichever comesfirst.

If any open time remains in the selected TSS, the remaining open time ofthe TSS is filled 652 with absolute placebo content. The algorithm ofFIG. 39A also involves randomly ordering 654 the content within the TSS.If the TSS contains any absolute placebo content, randomization ensuessuch that equal durations of the placebo content separate theexperimental content pieces.

Another open TSS is randomly selected 656 between the experimentstarting and ending points. A piece of experimental content that has notbeen assigned to a previously filled TSS is randomly selected 658. Ifall pieces of content have been assigned, absolute placebo content isselected. If absolute placebo content was selected in block 658, theselected TSS is filled 660 with absolute placebo content, otherwise theselected piece of experimental content is assigned to the selected TSS,and this TSS is filled in accordance with the processes of blocks646-654. Open TSSs continue to be filled according to the processes ofblocks 640-660 until all pieces of experimental content have beenassigned to a TSS.

FIG. 39B illustrates processes of an algorithm that assigns content totime-slot samples in accordance with embodiments of the presentinvention. The embodiment shown in FIG. 39B is directed to algorithmthat ensures that there are no location confounds during the duration ofinterest, after which the content is viewed not to be of interest if thecontent caused a change in the behavioral or transactional data beingmeasured. That is, the algorithm of FIG. 39B ensures that a viewer couldnot be exposed to one level of an independent variable and act on it ata different location that is testing a different level of theindependent variable during the duration of interest.

A potential drawback of using all experimental locations in such a wayas to eliminate all location confounds is that any location that is usedin this fashion is not able to be exposed to multiple levels of the sameindependent variable. As such, it would be difficult to measure howcombinations of different levels of an independent variable wouldinteract with one another within the same location. It may be desirable,under some circumstances, to first select a pre-determined number oflocations to be assigned experimental content for completewithin-location testing effects and then run this algorithm to use theremaining locations for testing without between-location confounds. Thatis, for example, one could use FIG. 38H to meet a pre-determined samplesize for within-location factors, and then use FIG. 39B to measure theeffects of content across locations.

The algorithm shown in FIG. 39B involves randomly selecting 670 anyexperimental location, and selecting 672 all locations related to theselected location. Content is randomly assigned 674 to the locationsselected in the preceding two blocks 670, 672 with the constraint thatonly unrelated content pieces are assigned to the locations. Anotherexperimental location is randomly selected 676 with the constraint thatit is unrelated to any locations already selected. All locations relatedto the location selected in the previous block, 676, and unrelated toselected locations for blocks 670 and 672 are selected 678. Content israndomly assigned 680 to the locations selected in the preceding twoblocks 676, 678 with the constraint that only unrelated content piecesare assigned to these locations. The processes of blocks 676-680 arerepeated until there are no unrelated locations remaining.

System according to embodiments of the present invention may include oneor more of the features, structures, methods, or combinations thereofdescribed herein. For example, systems may be implemented to include oneor more of the advantageous features and/or processes illustrated inFIGS. 40A-40C. It is intended that such systems need not include all ofthe features described herein, but may be implemented to includeselected features that provide for useful structures and/orfunctionality.

A digital signage system (DSS) according to embodiments of the presentinvention is shown in FIG. 40A. The DSS illustrated in FIG. 40A is acomputerized system configured to present informational content viaaudio, visual, and/or other media formats. The DSS may includefunctionality to automatically or semi-automatically generate playlists,which provide a list of the information content to be presented, andschedules, which define an order for the presentation of the content. Ina semi-automatic mode, a user may access a DSS control processor 405 viaan interactive user interface 410. Assisted by the DSS control processor405, the user may identify content to be presented and generateplaylists and schedules that control the timing and order ofpresentations on one or more DSS players 415. Each player 415 presentscontent to recipients according to a playlist and schedule developed forthe player. The informational content may comprise graphics, text, videoclips, still images, audio clips, web pages, and/or any combination ofvideo and/or audio content, for example.

In some implementations, after a playlist and schedule are developed,the DSS control processor 405 determines the content required for theplaylist, downloads the content from a content server, and transfers thecontent along with the playlist and schedule to a player controller 420that distributes content to the players 415. Although FIG. 40A showsonly one player controller 420, multiple player controllers may becoupled to a single DSS control processor 405. Each player controller420 may control a single player or multiple players 415. The contentand/or the playlists and schedules may be transferred from the DSScontrol processor 405 to the one or more player controllers 420 in acompressed format with appropriate addressing providing informationidentifying the player 415 for which the content/playlist/schedule isintended. In some applications, the players 415 may be distributed instores or malls and the content presented on the players 415 may beadvertisements.

In other implementations, the DSS control processor 405 may transferonly the playlists and schedules to the player controller 420. If thecontent is not resident on the player controller 420, the playercontroller 420 may access content storage 425 to acquire the content tobe presented. In some scenarios, one or more of the various componentsof the DSS system, including the content storage 425, may be accessiblevia a network connection, such as an intranet or Internet connection(wired or wireless). The player controller 420 may assemble the desiredcontent, or otherwise facilitate display of the desired content on theplayers according to the playlist and schedule. The playlists,schedules, and/or content presented on the players 415 can be modifiedperiodically or as desired by the user or automatically(algorithmically) through the player controller 420, or through the DSScontrol processor 405, for example.

In some implementations, the DSS control processor 405 facilitates thedevelopment and/or formatting of a program of content to be played on aplayer. For example, the DSS control processor 405 may facilitateformatting of an audiovisual program through the use of a template. Thetemplate includes formatting constraints and/or rules that are appliedin the development of an audiovisual program to be presented. Forexample, the template may include rules associated with the portions ofthe screen used for certain types of content, what type of content canbe played in each segment, and in what sequence, font size, and/or otherconstraints or rules applicable to the display of the program. Aseparate set of rules and/or constraints may be desirable for eachdisplay configuration. In some embodiments, formatting a program fordifferent displays may be performed automatically by the DSS controlprocessor 405.

In some embodiments, the DSS may create templates, generate content,select content, assemble programs, and/or format programs to bedisplayed based on information acquired through research andexperimentation in the area of cognitive sciences. Cognitive scienceseeks to understand the mechanisms of human perception. The disciplinesof cognitive and vision sciences have generated a vast knowledge baseregarding how human perceptual systems process information, themechanisms that underlie attention, how the human brain stores andrepresents information in memory, and the cognitive basis of languageand problem solving.

Application of the cognitive sciences to content design, layout,formatting, and/or content presentation yields information that iseasily processed by human perceptual systems, is easy to understand, andis easily stored in human memory. Knowledge acquired from the cognitivesciences and stored in a cognitive sciences database 430 may be usedautomatically or semi-automatically to inform one or more processes ofthe DSS including creation of templates, content design, selection ofcontent, distribution of content, assembly of programs, and/orformatting of programs for display. The cognitive sciences database 430used in conjunction with the programming of the DSS yieldsadvertisements or other digital signage programs that are enhanced bythe teachings of cognitive science, while relieving the system user fromneeding specific training in the field.

For example, cognitive sciences database 430 may store cognitive andvision science information that is utilized during the content design,distribution, and/or adjustment processes in order to provide contentthat is easily processed by human perceptual systems, easilycomprehended, and easily stored in memory. Cognitive sciences database430 may include design rules and templates that may be implemented by acomputer to develop and modify content in conformance with principles ofcognitive and vision sciences. Cognitive sciences database 430 may alsoinclude computer implementable models of principles of cognitive andvision sciences, such as models of visual attention, text readability,and memory principles.

In development of a digital signage program, e.g., ad campaign or thelike, the DSS control processor 405 may guide a user through variousprocesses that are enhanced using knowledge acquired through thecognitive sciences. For example, information stored in the cognitivesciences database 430 may be applied to the choice of templates toproduce an optimal program layout and/or to the selection of content,such as whether content elements should be graphical, text, involvemovement, color, size, and/or to the implementation of other aspects ofprogram development

Computer assisted methods and systems of the present invention may beimplemented to allow content designers, who typically do not have thetraining required to apply principles from cognitive science and visionscience, to increase the effectiveness of content design anddistribution. Systems and methods of the present invention mayincorporate features and functionality involving cognitive sciencesdatabase 430 in manners more fully described in co-pending U.S. patentapplication Ser. No. 12/159,106, filed on Dec. 29, 2006 as InternationalApplication US2006/049662 designating the United States entitled“Content Development and Distribution Using Cognitive SciencesDatabase,” which is incorporated herein by reference.

The DSS may include the capability for designing alternative versions ofa digital signage program to accommodate diverse display types andviewing conditions. Display technology is diverse and there are largedifferences in the types of displays used to present content on adigital signage network. For example, the size, shape, brightness, andviewing conditions will vary greatly across a digital signage network(e.g., some displays may be small, flexible and non-rectilinear, whereasothers may be standard large format Liquid Crystal Display (LCD) andplasma displays). The variation in display types and viewing conditionsmeans that any single version of a piece of content may not be optimalfor all the displays across a network.

In order to overcome this problem, it may be necessary to generateversions of each piece of content for each display type and viewingenvironment, and to selectively distribute these versions of content totheir corresponding screens in the network. However, it is not realisticto expect content designers to have such detailed knowledge of thedisplay types and viewing conditions across a large DSS network.Furthermore, even if such content designers had such detailed knowledge,it would be time-consuming to manually create versions of content foreach display and to manually schedule the content to play on eachcorresponding display at the appropriate time.

The DSS may include a data acquisition unit 435 for collecting data usedto improve the effectiveness of deployed content. The data acquisitionunit 435 allows distribution factors that underlie the effectiveness ofdigital signage networks to be continuously gathered in real-time duringdeployment of content. The information acquired can facilitatecontinuous improvement in content effectiveness of the DSS as well asimprovement of individual versions of content pieces. Previouslyacquired data may be used to learn what sensor or sales events shouldtrigger the display of specific types of content, for example.

Individual pieces of content in any content program each have a specificgoal (e.g., to sell a specific product). It is usually the case thatthere is variability in the value of each goal to the user of thedigital signage network. For example, there may be variability in theprofit margin and inventory level for each product which factor into thevalue of the goal for the product. The value of achieving each goalcontinuously changes during the time a digital signage program isdeployed. For example, the inventory level of a product may change, thusaffecting the goal for sales of the product.

Enhancing the effectiveness of a DSS as a whole, involves 1) accurateprediction of the impact of deploying a digital signage program on thegoal associated with the digital signage program, and 2) continuouslychanging the distribution patterns (timing, frequency, and location) ofindividual pieces of content as the value of each individual goalcorresponding to the pieces of content change. In many cases, it isunfeasible for users of the DSS to predict the impact of deployingcontent and to manually change content distribution patterns based oncontinuously changing values of goals associated with each piece ofcontent. The DSS provides the functionality to predict the impact ofdigital signage programs and to alter the distribution of content basedon the predictions.

As previously stated, content is displayed on the players 415 with thegoal of affecting human behavior (e.g., to impact purchasing behavior).However, prior digital signage systems are unable to demonstrate acause-and-effect relationship between signage content and human behavioror to measure the strength of the cause and effect relationship. Thisdifficulty arises because the methods by which content is deliveredacross current digital signage networks does not support thedetermination of whether any measured change in human behavior wascaused by signage content or the result of some confounding factors(e.g., change in weather, change in general demand for the product,change in price of the product).

The only way to decisively determine cause-and-effect relationshipsbetween signage content and human behavior is to conduct a trueexperiment during which signage content is systematically manipulatedusing complex experimental designs, and the effects of thosemanipulations on human behavior are carefully measured. Manuallyconducting such experiments is time consuming and requires significantknowledge and training in the scientific method of how to design trueexperiments. The users of digital signage systems may not havesufficient training to understand how to design a true experiment toacquire confound-free results.

The DSS illustrated in FIG. 40A includes a experiment design processor440 and user interface 410 that provide the capability to design trueexperiments, and also includes a machine learning design processor 439and the user interface 410 that provides the capability to designmachine learning routines. Also included in the DSS shown in FIG. 40A isan experiment deployment unit 445 configured to control execution ofcause-and-effect experiments and a machine learning deployment unit 437configured to control execution of machine learning routines.

FIG. 40B illustrates a system that is configured to design, conduct, andanalyze data for cause-and-effect experiments and machine learningroutines in accordance with embodiments of the invention. The systemillustrated in FIG. 40B includes an experiment design processor 440 thatis configured to design a true experiment or sub-processes that haveconstraints of a true experiment (e.g., such as those depicted in FIGS.37-39). As previously discussed, the experiment design processor 440 maybe configured to operate fully automatically or semi-automatically withuser interaction. In semi-automatic mode, the experiment designprocessor 440 may lead a user through various interactive sessionsconducted via the user interface 410 to design a true experiment. Insuch a process, the experiment design processor 440 ensures the designof a true experiment that produces confound-free data. Thus, a user isable to rely on the programming of the experiment design processor 440and is not required to have knowledge or experience in designing trueexperiments. The DSS may comprise only an experiment design processor440, or may include additional elements such as an experiment deploymentunit 445, a data acquisition unit 435, and data analysis unit 450. Thesystem shown in FIG. 40B also includes a machine learning designprocessor 439 that is configured to facilitate the design of one or moremachine learning routines.

The system may further include an experiment deployment unit 445. Theexperiment deployment unit 445 is configured to facilitate deployment ofthe experiment. In the context of a representative DSS system, theexperiment deployment unit 445 formats the experimental content and thecontrol group content for various player configurations and facilitatesthe transfer of the experimental content and the control content to theplayer controller 420 for presentation on players 415 as specified bythe playlists and schedules. A machine learning deployment unit 437 ofthe DSS system coordinates execution of one or more machine learningroutines, such as those discussed above, and formats content to be usedby the MLRs. The machine learning deployment unit 437 facilitates thetransfer of MLR content to the player controller 420 for presentation onplayers 415 as specified by MLR playlists and schedules.

The data acquisition unit 435 may be configured to collect experimentaldata from the control and treatment groups and optimization data for themachine learning routine. The data acquisition unit 435 may perform orfacilitate acquisition of data associated with the experiment and themachine learning routine via any means. For example, in the context ofthe exemplary DSS, the data acquisition unit 435 may be coupled tovarious sensor or data acquisition devices 462, 464, 466 that gatherinformation including product movement, product sales, customer actionsor reactions, and/or other information. Sensors 462 may be used todetect, for example, if a customer picks up the product, or if acustomer is in the vicinity of the display when the content isdisplayed. Sales may be determined based on information acquired by apoint of sales (POS) system 464. One or more devices 466 that validatethe display of content may also be used. Changes in inventory levels ofa product may be available via an inventory control system. Customerreactions may be acquired via questionnaires. If the conductedexperiment is a true experiment, the data acquired by the dataacquisition unit 435 is substantially confound-free.

The data acquisition unit 435 may be coupled to a data analysis unit 450that is configured to analyze the experimental data collected by thedata acquisition unit 435. The data analysis unit 450 may determineand/or quantify cause and effect relationships between the independentand dependent variables of the experiment. For the illustrated DSS, theresults of the analysis may be used to determine if the content iseffective at influencing product sales. The data analysis unit 450 mayanalyze acquired data for purposes of optimizing content distributionpatterns that maximize one or more effectiveness metrics, such aspoint-of-purchase sales, upgrades, and customer loyalty, and the like.

Because the analysis unit 450 will have received information regardingthe independent and independent variables (e.g., whether the IVs and DVsare continuous or discrete), the analysis unit 450 would have much ofthe necessary information to choose the appropriate statistical test toapply to the data from the experiment. For example, if there is one IVwith two discrete levels and one continuous DV, then a T-Test would beused for the inferential statistical test whereas if there is one IVwith two discrete levels and one DV with two discrete levels, then aChi-Squared test would be used for the inferential statistical test.Likewise, because analysis unit will access to information from thedesign processor 440 regarding which experimental conditions arediagnostic of particular hypotheses, the analysis unit 450 would havemost or all of the information needed to determine which experimentaland control conditions should be statistically compared.

The results of these analyses may be additionally or alternatively usedto implement or modify various processes. For example, if the contentwas effective at influencing product sales, an advertisement campaignmay be developed incorporating the content. A value may be assigned tothe content by a content valuation process 472 based on theeffectiveness of increasing sales. An advertiser using the content maybe invoiced by a billing unit 474 according the value of the content.The data analysis unit 450 may also provide information to inventorycontrol 476. Additionally, the data analysis unit 450 may provideinformation to a sales prediction unit 478 that generates a predictionof sales when the advertising campaign is deployed. The sales predictionunit 478 may additionally or alternatively predict the product inventoryneeded to support the sales generated by the advertisement campaign.

Implementation of a digital signage system, including capabilities forgenerating digital signage content, deploying experiments designed bythe expert system, and collecting experimental data are furtherdescribed in co-pending U.S. patent application Ser. No. 11/321,340filed Dec. 29, 2005 and in U.S. patent application Ser. No. 12/159,107filed on Dec. 29, 2006 as International Application US2006/049657, andentitled “Expert System for Designing Experiments,” which areincorporated herein by reference.

The systems and methods described herein may form the basis of aconsulting business according to embodiments of the present invention.Services offered could include, but not be limited to, working withcustomers to characterize their time-slot samples as appropriate forcertain communication objective and certain consumer audiences,determining which variables a study would address, determining levels ofindependent variables for testing, determining factors that could beused for blocking and randomizing, and conducting a power analysis,among others. A measurement algorithm as previously described may beused to specify time-slot allocation requirements for cross-optimizationand blocking factors.

Another application in accordance with the present invention is directedto systems and method for maximizing overall profitability. A system ofthe present invention may be used to optimize allocation of allavailable time-slot samples for two objectives: (1) contenteffectiveness testing as described in detail hereinabove, and (2)content that is not being tested but meant to address any number ofbusiness goals, such as increasing sales, promoting consumersatisfaction, informing employees, etc.

A system implemented according to the present invention as describedherein may provide the data to “balance” the total inventory oftime-slot samples, allowing the user to determine optimal levels oftesting versus non-testing time-slot samples, and allocations withinthose groups to more efficiently test content using the minimal numberof time-slot samples, freeing more time-slot samples for non-testingcontent. Results data could inform users as they seek to continuouslymonitor and adjust content distribution to maximize profitability,satisfaction, etc. and could aid users in determining when content is“worn-out,” defined as the point in time when previously effectivecontent ceases to be sufficiently effective due to over-exposure to theconsumer or employee audience.

FIG. 40C is a diagram of a digital signage network that includes variouscomponents of a DSS in accordance with embodiments of the presentinvention. According to FIG. 40C, the DSN includes a DSN system module429 communicatively coupled to an experiment system 423 and a machinelearning system 427. Also communicatively coupled to the DSN systemmodule 429 is a server 421, which cooperate to control contentdistribution across the DSN.

The DSN system module 429 is configured to distribute content across thenetwork, depicted as DSN infrastructure in FIG. 40C, and collect andprocess various data. The DSN system module 429 cooperates with theexperiment system 423 to conduct cause-and-effect experiments, and withthe machine learning system 427 to execute machine learning routines.The DSN system shown in FIG. 40C may exclude the experiment system 423for those embodiments that execute machine learning routines exclusiveof running experiments.

The DSN system module 429 may comprise a top-level decision tool, suchas that described hereinabove in the context of various embodiments(e.g., see FIG. 30). For example, the DSN system module 429 may beconfigured to implement algorithms that continuously monitor the networkand decide how to allocate control of each time period (e.g., TSS) foreach display to the subcomponent systems (a) the experiment system 423and (b) the MLR system 427. As previously discussed, the decision tooluses inputs from: the user regarding the value of experimental insights,the experiment system regarding the required sample size/duration tomeet the desired statistical power, the incoming dependent variable dataas the experiment progresses, and the estimated or known value ofallowing the MLR to control the TSSs in order to maximize currentbusiness goals. It is understood that the decision tool may beimplemented in components of the DSN system other than the DSN systemmodule 429 (e.g., the experiment system 423, MLR system 427), and may bedistributed among various components.

The DSN system module 429 communicates with a multiplicity of displays415 via the DSN infrastructure 413, which may include one or both ofwired and wireless networks. The DSN infrastructure 413 shown in FIG.40C incorporates one or more mobile networks 417 and one or more datanetworks 419. The mobile network(s) 417 may represent any one or moreknown or future wireless networking technologies, such as the GlobalSystem for Mobile Communications (GSM), Universal MobileTelecommunications System (UMTS), Personal Communications Service (PCS),Time Division Multiple Access (TDMA), Code Division Multiple Access(CDMA), Wideband CDMA (WCDMA), or other mobile network transmissiontechnologies. One or more data networks 419 may cooperatively operatewith the mobile network(s) 417 (or operate exclusive of mobilenetwork(s) 417) to facilitate data transfers to and from the DSN systemmodule 429. For example, the illustrated data network 419 may representthe Internet, which interfaces to the illustrated mobile network 417 toprovide landline connectivity with the DSN system module 429.

In some embodiments, sets of displays 415 are coupled to one or moreplayer controls 420 which communicate with the DSN system module 429 viathe DSN infrastructure 413. The connections between a player control 420and the displays 415 and DSN infrastructure 413, respectively, may bewired, wireless or a combination of wired and wireless connections. Inother embodiments, a player control 420 need not be used to serve as aninterface between the displays 415 and the DSN infrastructure 413.Content distribution and data acquisition may be managed using astreaming technology that allows the DSN system module 429 to coordinateand execute playlist schedules for a multiplicity of displays 415without the player control 420. Suitable transport approaches includeautomatic retry query (ARQ), TCP, and UDP streaming, among others.

Using the description provided herein, embodiments of the invention maybe implemented as a machine, process, or article of manufacture by usingstandard programming and/or engineering techniques to produceprogramming software, firmware, hardware or any combination thereof.

Any resulting program(s), having computer-readable program code, may beembodied on one or more computer-usable media such as resident memorydevices, smart cards, DVDs, CD, or other removable memory devices, ortransmitting devices, thereby making a computer program product orarticle of manufacture according to the invention. As such, the terms“article of manufacture” and “computer program product” as used hereinare intended to encompass a computer program that exists permanently ortemporarily on any computer-usable medium or in any transmitting mediumwhich transmits such a program.

The foregoing description of the various embodiments of the inventionhas been presented for the purposes of illustration and description. Itis not intended to be exhaustive or to limit the invention to theprecise form disclosed. Many modifications and variations are possiblein light of the above teaching. For example, embodiments of the presentinvention may be implemented in a wide variety of applications. It isintended that the scope of the invention be limited not by this detaileddescription, but rather by the claims appended hereto.

What is claimed is:
 1. A computer system having one or more processorsand memories, comprising: an experiment module, executed on the one ormore processors, configured to conduct an experiment, and useexperimental content to determine effectiveness of communication contentin accordance with a schedule comprising a first set and a second set oftime periods, the experiment conducted using the first set of timeperiods; and a machine learning routine (MLR) module, executed on theone or more processors, configured to execute, while conducting theexperiment, a machine learning routine, and use MLR content to enhancean effectiveness metric in accordance with the schedule, wherein: themachine learning routine comprises one or both of an explore routineassociated with explore content and an exploit routine associated withexploit content; for experimental content that are related to either ofexplore or exploit content, using the second set of time periods for theexplore or exploit routine; and for experimental content that areunrelated to either of explore or exploit content, using at least someof the first set of time periods for the explore or exploit routine. 2.The computer of claim 1, further comprising: a module for displaying theexperimental content and the MLR content on a display according to theschedule.
 3. The computer system of claim 1, wherein the machinelearning routine comprises a reinforcement learning routine, a logisticregression routine, an unsupervised learning routine, a semi-supervisedlearning routine, or use of one or more neural networks.
 4. The computersystem of claim 1, wherein the MLR module is operative to identify afrequency rate for presenting the explore content and the exploitcontent that results in enhancement of the predetermined effectivenessmetric.
 5. The computer system of claim 1, further comprising: anevaluation module, executed on the one or more processors, configured toperform an evaluation to determine, for any given time period, if usingexperimental content has more value than using MLR content for the timeperiod.
 6. The computer system of claim 1, further comprising: aschedule module, executed on the one or more processors, configured togenerate the schedule, wherein the module receives one or more of (a) avalue of the experimental hypothesis or hypotheses, (b) a value of eachcategory of effectiveness metric, (c) viewer visit duration (VVD) foreach business objective, (d) content restrictions, and (e) urgencyinformation, and wherein the schedule module generates the schedule inaccordance with one or more of (a) through (e).
 7. The computer systemof claim 6, wherein the schedule module continuously adjusts theschedule in terms of times and display locations to maximize at leastone effectiveness metric.
 8. The computer system of claim 1, furthercomprising: an analysis module, executed on the one or more processors,configured to analyze measurement data indicative of MLR contenteffectiveness for the machine learning routine, and store themeasurement data in a historical database.
 9. The computer system ofclaim 1, further comprising: an output module, executed on the one ormore processors, configured to produce output data indicative of returnon investment for each piece of MLR content based on one or both ofdaypart and display location.
 10. A computer-implemented methodcomprising: conducting, by a processing system, an experiment usingexperimental content to determine effectiveness of communication contentin accordance with a schedule comprising a first set and a second set oftime periods, the experiment conducted using the first set of timeperiods; and executing, by the processing system, while conducting theexperiment, a machine learning routine (MLR) using MLR content toenhance an effectiveness metric in accordance with the schedule,wherein: the machine learning routine comprises one or both of anexplore routine associated with explore content and an exploit routineassociated with exploit content; for experimental content that arerelated to either of explore or exploit content, using the second set oftime periods for the explore or exploit routine; and for experimentalcontent that are unrelated to either of explore or exploit content,using at least some of the first set of time periods for the explore orexploit routine.
 11. The computer-implemented method of claim 10,further comprising: displaying experimental content and MLR content on adisplay according to the schedule.
 12. The computer-implemented methodof claim 10, wherein the machine learning routine comprises areinforcement learning routine, a logistic regression routine, anunsupervised learning routine, a semi-supervised learning routine, oruse of one or more neural networks.
 13. The computer-implemented methodof claim 10, further comprising identifying a frequency rate forpresenting the explore content and the exploit content that results inenhancement of the predetermined effectiveness metric.
 14. Thecomputer-implemented method of claim 10, further comprising: performing,by the processing system, an evaluation to determine, for any given timeperiod, if using experimental content has more value than using MLRcontent for the time period.
 15. The computer-implemented method ofclaim 10, further comprising: receiving one or more of (a) a value ofthe experimental hypothesis or hypotheses, (b) a value of each categoryof effectiveness metric, (c) viewer visit duration (VVD) for eachbusiness objective, (d) content restrictions, and (e) urgencyinformation; and generating, by the processing system, the schedule forimplementing the experiment and machine learning routine in accordancewith one or more of (a) through (e).
 16. The computer-implemented methodof claim 15, further comprising: continuously adjusting the schedule interms of times and display locations to maximize at least oneeffectiveness metric.
 17. The computer-implemented method of claim 10,further comprising: analyzing, by the processing system, measurementdata indicative of MLR content effectiveness for the machine learningroutine, and storing the measurement data in a historical database. 18.The computer-implemented method of claim 10, further comprisingproducing output data indicative of return on investment for each pieceof MLR content based on one or both of daypart and display location. 19.A computer system having one or more processors and memories,comprising: a module executed on the one or more processors configuredto conduct an experiment and use experimental content to determineeffectiveness of communication content in accordance with a schedulecomprising a first plurality of time-slot samples and a second pluralityof time-slot samples, the experiment being conducted using the firstplurality of time-slot samples; and a module executed on the one or moreprocessors configured to execute, while conducting the experiment, amachine learning routine (MLR) and use MLR content to enhance aneffectiveness metric in accordance with the schedule, wherein: themachine learning routine comprises a reinforcement learning routine, thereinforcement learning routine comprising one or both of an exploreroutine associated with explore content and an exploit routineassociated with exploit content; for experimental content that arerelated to either of explore or exploit content, using the secondplurality of the time-slot samples for the explore or exploit routine;and for experimental content that are unrelated to either of explore orexploit content, using at least some of the first plurality of time-slotsamples for the explore or exploit content.